It looks like you could start by trying to run ./scripts/build.sh and ./scripts/deploy.sh by hand separately to see which script causes the error.
Then if that works, you can insert echo commands to tell you what part of whichever script breaks.
Back. So I couldn’t figure out how to just “run” a .sh file and it sounded like maybe I had to figure out “installing bash”, so I installed wlm or something… Looks like it installed Ubuntu, unfortunately I’m out of my depth… running “bash” from command line actually does something now anyways.
When I try to use bash and the build.sh (which is what I think you’re suggesting) I get the below. Is that helpful? It looks like stuff is missing, but it’s possible that stuff is done before this script is actually run? No idea what yarn run deploy actually does…
PS C:\Users\stobot> cd tw5-firebase
PS C:\Users\stobot\tw5-firebase> bash ./scripts/build.sh
: invalid optionsh: line 2: set: -
set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
: No such file or directory /mnt/c/Users/stobb/tw5-firebase/scripts
./scripts/build.sh: line 6: $'\r': command not found
./scripts/build.sh: line 7: pushd: $'/mnt/c/Users/stobb/tw5-firebase/scripts\r/..\r': No such file or directory
./scripts/build.sh: line 8: $'\r': command not found
: No such file or directory /mnt/c/Users/stobb/tw5-firebase/scripts
./scripts/build.sh: line 10: $'\r': command not found
: No such file or directory: /mnt/c/Users/stobb/tw5-firebase/scripts
./scripts/build.sh: line 12: $'\r': command not found
: No such file or directory: /mnt/c/Users/stobb/tw5-firebase/scripts
./scripts/build.sh: line 14: $'\r': command not found
: No such file or directory: /mnt/c/Users/stobb/tw5-firebase/scripts
./scripts/build.sh: line 16: $'\r': command not found
./scripts/build.sh: line 17: $'popd\r': command not found
We must solve this. As far as I can see you could use bob if it can be hosted on the internet safely in a docker container or VM but these are not available to us all.
Alternatively a LAN resource can be accessed securely if the team has a VPN into the LAN. The harder part is when a team member is outside the LAN without a VPN etc…
2 Likes
Oh! I didn’t catch that you were using PS. All that speed reading.
I think you just need to go to your firebase directory and
yarn ./scripts/build.sh
PS C:\Users\stobot\tw5-firebase\scripts> yarn build.sh
yarn run v1.22.17
error Command "build.sh" not found.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
makes it seem like the file is not there… but
PS C:\Users\stobot\tw5-firebase\scripts> dir
Directory: C:\Users\stobot\tw5-firebase\scripts
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 11/10/2021 9:15 AM 552 backup2.sh
-a---- 11/10/2021 9:15 AM 452 build-backend.sh
-a---- 11/10/2021 9:15 AM 269 build-cli.sh
-a---- 11/10/2021 9:15 AM 881 build-frontend.sh
-a---- 11/10/2021 9:15 AM 148 build-jsonschema.sh
-a---- 11/10/2021 9:15 AM 452 build-prerequisites.sh
-a---- 11/10/2021 9:15 AM 278 build-tests.sh
-a---- 11/10/2021 9:15 AM 268 build.sh
-a---- 11/10/2021 9:15 AM 483 cli.sh
-a---- 11/10/2021 9:15 AM 308 deploy.sh
-a---- 11/10/2021 9:15 AM 1515 env.sh
-a---- 11/10/2021 9:15 AM 312 export.sh
-a---- 11/10/2021 9:15 AM 146 firebase.sh
-a---- 11/10/2021 9:15 AM 652 gcloud-shell.sh
-a---- 11/10/2021 9:15 AM 530 gettoken.sh
-a---- 11/10/2021 9:15 AM 157 logs.sh
-a---- 11/10/2021 9:15 AM 1262 renderdoc.sh
-a---- 11/10/2021 9:15 AM 397 repl.sh
-a---- 11/10/2021 9:15 AM 230 run-local-backend.sh
-a---- 11/10/2021 9:15 AM 528 run-local-frontend.sh
-a---- 11/10/2021 9:15 AM 308 stripnewline.sh
-a---- 11/10/2021 9:15 AM 318 test.sh
-a---- 11/10/2021 9:15 AM 334 tiddlers-to-firestore.sh
-a---- 11/10/2021 9:15 AM 257 tiddlers-to-fs.sh
-a---- 11/10/2021 9:15 AM 265 update-supported-browsers.sh
I can see it’s clearly there… I’m sure I’m missing something embarassingly simple. Also, if using something other than PS, happy to try that.
Thanks @TW_Tones , yes have used BOB in the past but about once a day someone did something that crashed it. PLUS nobody could easily access it via phone being local IP.
I have to admit that I’m sitting here considering normal Node.js with host=0.0.0.0 despite the warnings just to try it… doesn’t solve phone, but probably more stable than BOB for me…
1 Like
Sounds like you set up the Windows Subsystem for Linux… which is good. I think there is an above average chance your problems might be related to needing to execute the scripts from bash.
I am not a Windows expert but I think it might be worth adding Ubuntu’s bash to the terminal in Windows and then running the scripts.
Once you have that, I am not sure if the option below to open a tab with the Ubuntu bash shell comes up as default or if you need to configure it:
This is almost hilarious. Yes I’d installed Ubuntu and was using Terminal already. Got in there (see screenshot) and then it looks like yarn was no longer installed so tried to install it via npm, but that wasn’t in there and googling said to try installing nodejs in there, but got stuck again…
Different approach. Yarn probably sets up dependencies and variables. So edit the bottom of build.sh with something like
echo “At bottom of build.sh”
And then run yarn deploy again. If it gets to the message, then you know to move to deploy.sh. Put in marker echo messages until you narrow down what line is causing the problem.
Alternatively, you can edit package.json and change what ‘deploy’ runs so it just runs ‘build.sh’ .
You’ve replied to @stobot 3 times, did you know you could send them a personal message instead?
Did you know that Discourse is not human? Did you? Hm? Hm? Grr.
Strangely even with the echo something opens for a second and then goes away…
The build.sh file now looks like:
#!/usr/bin/env bash
set -e
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
# set up env
. "$DIR/env.sh"
pushd "$DIR/.."
"$DIR/build-jsonschema.sh"
"$DIR/build-frontend.sh"
"$DIR/build-backend.sh"
"$DIR/build-cli.sh"
popd
echo "at bottom of build.sh"
but when I do the yarn run deploy bit (in PowerShell) it acts the same
PS C:\Users\stobb\tw5-firebase> yarn run deploy
yarn run v1.22.17
$ ./scripts/build.sh && ./scripts/deploy.sh
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
thanks @Mark_S - I’m going to have to give up for the night now, appreciate you taking the time - I’ll try a bit more tomorrow.
So now you’re running everything in the bash shell (which means under the covers in the Ubuntu install), so you need to install things in Ubuntu. I suspect the install scripts for the project you are trying to use assume a *nix OS and therefore are having issues.
In the bash shell try this:
sudo apt update
sudo apt upgrade
sudo apt install build-essential
sudo apt install nodejs npm
sudo apt install jq
npm install --global yarn
yarn install #in the project dir
That probably means the error is occurring in one of these:
"$DIR/build-jsonschema.sh"
"$DIR/build-frontend.sh"
"$DIR/build-backend.sh"
"$DIR/build-cli.sh"
You can put echos before each of these to see how far it gets.
2021 and developers are still giving us single digit error codes!
It’s Google Cloud, a service provider. Unlike many other things – you’re directly paying Google for a service, just like you might buy hosting from Amazon Web Services or Microsoft’s Azure cloud.
What I’m working on with Fission I hope to eventually be a competitor to the all-in-one Firebase stack, which is the entire backend that many custom apps use. Which, of course, locks them into Google’s cloud, since it’s proprietary to them.
But I don’t have any Google-is-tracking-me concerns in this case, they’re just a service provider.
This does look like a good solution. The other similar solution Tidly that I’ve run used Google’s App Engine (which is a bit weird and old).
There are some other open source Firebase competitors like Supabase https://supabase.io/ that could fit here.
But, to the task at hand for @stobot – a multi user, online hosted TW with everything you need would be a magical thing for many communities and teams. Thanks for bringing this up!
1 Like
@saqimtiaz - ran all of those and they worked fine, then tried the yarn run deploy while still in umbuntu and got this:
stobbea@STOBBE-LAPTOP:/mnt/c/Users/stobb/tw5-firebase$ yarn run deploy
yarn run v1.22.17
$ ./scripts/build.sh && ./scripts/deploy.sh
/usr/bin/env: ‘bash\r’: No such file or directory
error Command failed with exit code 127.
different error code anyways. Something about this bash\r “no such file or directory” seems like a clue…
Did you do a complete install of linux on windows, like perhaps in this tutorial ? Also there’s a comment at the bottom of one of these tutorials that you don’t get the bash terminal until a reboot, so that might be worth trying.
However, I don’t think that’s it. It’s saying ‘bash\r’ and not ‘bash’ which suggests to me that it doesn’t know how to interpret the line endings. Linux and Windows have different methods of terminating lines. Usually it somehow works.
I still think it’s worthwhile to try out the echo technique to see where the error is occurring, since all your instructions work through yarn.
Bash btw is the “Bourne again shell”, which was hilarious in 1972. It’s pretty standard for it to be installed as the default shell. You can test if it’s installed just by typing and entering “bash” from your terminal. If you get an error, then it’s probably not installed. If it looks like nothing happened, then you’re actually in the new bash shell (type “exit” and enter and you should be back in the terminal).
My first instinct is the same as Mark, it seems like a line ending problem.
First possibility is that when you edited the script to add the echo line, it saved with Windows line endings. So try restoring that file with git restore.
If that still does not resolve it, I suggest making a fresh git clone from the bash commandline, which might entail first installing git.
Hi guys!
Thanks @stobot for letting me know about this thread! I’ll definitely look into the issue you had with installation next week (this week is a bit busy).
First of all, thank you guys for trying out / discussing tw5-firebase! It gives me a warm feeling that others find it useful (or at least the concept interesting).
TW5-firebase is an ambitious project. Unfortunately I’ve had to put it on hold for the last couple of months, but I do plan to continue.
TL;DR: Jump to “Current Status”
Let me give you some insight into the whys and hows.
What’s the point?
A lot of us really like TiddlyWiki as a personal notebook. I’ve used Evernote, Joplin, etc. and they all have their strong points, but none of them offer the level of extensibility that TiddlyWiki does. In my opinion there’s several reasons for this:
- Everything is a tiddler - Macros, plugins, themes (what’s traditionally code) can be added to the wiki the same way articles and images (content).
- Anything can be (and is meant to be) customized. Any shadow tiddler can be overridden, no exceptions (icons, text - even the wikitext parsing logic).
- The entire UI is constructed out of TiddlyWiki macros. It’s turtles all the way down. The macro syntax can be a bit confusing initially, but it’s a lot more non-dev friendly than javascript. Combined with the previous points, this means one could in theory change anything they want about the way tiddlywiki works.
The result is a substrate that can be used to build all sorts of text-centric applications (TiddlyMap, GTD tiddlywikis, etc). As tiddlywiki provides a nice text editing experience, the application can be used to modify itself in a relatively non-programmer friendly way (eg: using tags to make a new menu item appear). Users must understand the specific tiddlywiki well, but not necessarily programming in general. In such an application adding a button or fixing a typo does not require a software developer’s participation.
The majority of text-based applications (including the original wiki functionality) become more valuable when collaboration is possible. TiddlyWiki has grown to become a fantastic tool as a personal wiki. Imagine the possibilities if we could make it a collaborative experience!
The road to collaboration
I believe a theoretical collaborative multi-user TiddlyWiki’s ideal users are small teams (less than a dozen individuals). Even in such a small, relatively high-trust scenario, certain guarantees are needed:
- Editing a tiddler should not ever result in accidentally overwriting another user’s simultaneous changes.
- It shouldn’t be possible for a user of the wiki to run un-sandboxed code in other users’ browsers (more on this in a bit).
- Administrators of the wiki should be able to add / remove users.
- There should be some form of access control (not everyone should be able to read / edit every tiddler).
- The complexity and cost of installing and operating the wiki should be modest.
It would be great if we could use an existing system for storing tiddlers. Unfortunately, I didn’t find anything that worked. The first point is difficult to achieve if the entire wiki is a single file. Solutions like the -otherwise excellent- TiddlyDrive https://tiddlydrive.github.io/ and TiddlyWiki in the Sky https://twcloud.github.io/tw5-dropbox/ thus don’t scale to the dozen-user teams I targeted. Storing each tiddler as a separate file in Google Drive or Dropbox would lead to very slow startup times and edit conflicts would be difficult to handle.
Collaborative web apps traditionally store their state in a transactional database to solve the problems around simultaneous user actions. Unfortunately installing and operating traditional web apps has been complex and time-consuming.
Firebase as a user-friendly backend
Installing and operating a database like postgres is not something a non-developer should have to do. Building on supabase as @boris mentioned is indeed an option, but at the time I started tw5-firebase, it wasn’t yet announced. Even today, it still lacks many of the components offered by firebase. Fission might be the perfect platform for a collaborative tiddlywiki server someday, but until then, we have Firebase.
Firebase attempts to provide solutions for all the traditional needs of a collaborative web apps while providing user-friendly dashboards. Namely:
- Authentication (firebase auth integrates with over a dozen identity providers)
- Static asset hosting (with integrated CDN)
- Database (firestore)
- User file hosting (different from static asset hosting, uses google storage)
- Server-side functions (run server-side javascript without managing servers)
All this is bundled up and nicely integrated. The various components know about each other. Firebase is well documented, with plenty of examples.
Of course, there are disadvantages:
- I’m just as disillusioned with Google as everyone else. The original “do no evil” mantra has seemingly disappeared - perhaps unavoidable at their scale, but still disappointing.
- Cloud functions requires users to opt into billing by providing their credit card. It gave me a pause too, I’m sure a lot of people would hesitate to proceed at that point.
- Firestore is not my favorite database. It’s impossible to run the real thing locally during development, performance is hard to analyze (no
EXPLAIN). The backup / restore tools are pretty lame. The way indexes work is hacky. There’s a surprisingly low ceiling to document size. It’s not horrible, but it’s not great either. - Cloud Functions are the only supported way of executing custom server-side logic, which means that long-running processes with lots of in-memory state don’t work. This brings us to our next point:
Why not use the built-in node.js server?
TiddlyWiki5 comes with a built-in webserver. Projects like Bob https://github.com/OokTech/TW5-Bob build on it. It works really well locally. Unfortunately, as I mentioned at the end of the previous section, it’s architecture is not very firebase friendly. On startup, the TW5 node app reads all tiddlers. There is no further reading of tiddlers from the filesystem. Once loaded, the “source of truth” for tiddlers is the node process’ memory.
If we were to naively move this to Firebase (and swap the filesystem for firestore), serving every single request would require reading all tiddlers before it even got to processing the request itself. Also, firebase cloud functions run in parallel, which would lead to users accidentally overriding each others’ edits. So we need something else.
Note that this isn’t really a firebase-specific issue: collaborative web apps generally have a central ACID-compliant database storing application state, and -potentially several- stateless web server processes running the app’s code. The web server processes can (and in the case of Cloud Functions do) get replaced frequently; the contents of their memory should not need to survive multiple requests. AWS Lambda, supabase, etc would all require following the same architecture.
The built-in node.js server is built the way it is because it’s meant to be super-compact, treating node.js as just another execution environment. It’s brilliant but doesn’t scale. Since TiddlyWiki is designed to be a personal wiki, it doesn’t need to.
Current TW5-firebase architecture
The current architecture (what’s currently in master:
- A static HTML file containing a very minimal customized TiddlyWiki loads and authenticates the user using firebase auth.
- Once authenticated, tiddlers are read from firestore via HTTP request to a firebase function.
- Updating tiddlers also happens via HTTP requests to firebase functions (which then write to firestore).
- Importing non-text tiddlers uploads them to firestore storage. A tiddler with a
_canonical_urifield points to the real URL of the uploaded file.
This is already useful, but it doesn’t do all that’s required for collaboration:
- No user-friendly way to add users (although a CLI tool for this is included).
- No user-friendly way to set permissions. Access control is managed by placing tiddlers in
bags. Individual tiddlers don’t have rights. Moving tiddlers between bags should be easy for a user and atomic. They’re not. - Currently, only admin users can add / update code in the wiki to protect users against other (malicious / playful) users. The wiki has access to firebase credentials, however, so an admin could impersonate the user and delete tiddlers in another wiki they have access to.
Next Architecture
- Firestore supports listening to changes in the DB. This means when one user makes an edit to a tiddler (and save the draft), any other user editing the same tiddler could be instantly modified. They would “merge” their versions of the tiddler before the second user saves, hopefully eliminating most “conflict on save” situations.
- In order to isolate user code within the wiki from firebase credentials, the wiki itself should run within a sandboxed iframe, while the “outer” frame communicates with firebase.
Current Status
In short, TW5-firebase is a (mostly) working prototype. It shouldn’t eat your data, but it is definitely not read for collaboration.
The master branch of tw5-firebase should work. I plan on fixing the issues @stobot faced with installation. Please note that the optimistic locking logic is not perfect: it does occasionally detect conflicts when it shouldn’t. This is not handled well by the current UI (the “save error” dialog appears). They can be resolved by deleting the draft tiddlers in question, but it’s not pretty.
The tiddler-version-manager at https://github.com/neumark/tw5-firebase/tree/tiddler-version-manager branch is an unfinished attempt at introducing the changes I mentioned in the “next architecture” section. I hope to continue someday, but haven’t been able to work on it for the last couple of months.
PS: I’m a bit upset that I’m only allowed to post 2 links!
2 Likes
I suspect that is because you are new to the forum. Hopefully one of the @moderators can bump up your level.
@stobot You can not run bash scripts with PowerShell. bash is unix and PowerShell is windows. … That doesn’t work!