Anyone using Firebase?

#22

This does look like a good solution. The other similar solution Tidly that I’ve run used Google’s App Engine (which is a bit weird and old).

There are some other open source Firebase competitors like Supabase https://supabase.io/ that could fit here.

But, to the task at hand for @stobot – a multi user, online hosted TW with everything you need would be a magical thing for many communities and teams. Thanks for bringing this up!

1 Like
#23

@saqimtiaz - ran all of those and they worked fine, then tried the yarn run deploy while still in umbuntu and got this:

stobbea@STOBBE-LAPTOP:/mnt/c/Users/stobb/tw5-firebase$ yarn run deploy
yarn run v1.22.17
$ ./scripts/build.sh && ./scripts/deploy.sh
/usr/bin/env: ‘bash\r’: No such file or directory
error Command failed with exit code 127.

different error code anyways. Something about this bash\r “no such file or directory” seems like a clue…

#24

Did you do a complete install of linux on windows, like perhaps in this tutorial ? Also there’s a comment at the bottom of one of these tutorials that you don’t get the bash terminal until a reboot, so that might be worth trying.

However, I don’t think that’s it. It’s saying ‘bash\r’ and not ‘bash’ which suggests to me that it doesn’t know how to interpret the line endings. Linux and Windows have different methods of terminating lines. Usually it somehow works.

I still think it’s worthwhile to try out the echo technique to see where the error is occurring, since all your instructions work through yarn.

Bash btw is the “Bourne again shell”, which was hilarious in 1972. It’s pretty standard for it to be installed as the default shell. You can test if it’s installed just by typing and entering “bash” from your terminal. If you get an error, then it’s probably not installed. If it looks like nothing happened, then you’re actually in the new bash shell (type “exit” and enter and you should be back in the terminal).

#25

My first instinct is the same as Mark, it seems like a line ending problem.

First possibility is that when you edited the script to add the echo line, it saved with Windows line endings. So try restoring that file with git restore.

If that still does not resolve it, I suggest making a fresh git clone from the bash commandline, which might entail first installing git.

#26

Hi guys!

Thanks @stobot for letting me know about this thread! I’ll definitely look into the issue you had with installation next week (this week is a bit busy).
First of all, thank you guys for trying out / discussing tw5-firebase! It gives me a warm feeling that others find it useful (or at least the concept interesting).

TW5-firebase is an ambitious project. Unfortunately I’ve had to put it on hold for the last couple of months, but I do plan to continue.
TL;DR: Jump to “Current Status”
Let me give you some insight into the whys and hows.

What’s the point?

A lot of us really like TiddlyWiki as a personal notebook. I’ve used Evernote, Joplin, etc. and they all have their strong points, but none of them offer the level of extensibility that TiddlyWiki does. In my opinion there’s several reasons for this:

  1. Everything is a tiddler - Macros, plugins, themes (what’s traditionally code) can be added to the wiki the same way articles and images (content).
  2. Anything can be (and is meant to be) customized. Any shadow tiddler can be overridden, no exceptions (icons, text - even the wikitext parsing logic).
  3. The entire UI is constructed out of TiddlyWiki macros. It’s turtles all the way down. The macro syntax can be a bit confusing initially, but it’s a lot more non-dev friendly than javascript. Combined with the previous points, this means one could in theory change anything they want about the way tiddlywiki works.

The result is a substrate that can be used to build all sorts of text-centric applications (TiddlyMap, GTD tiddlywikis, etc). As tiddlywiki provides a nice text editing experience, the application can be used to modify itself in a relatively non-programmer friendly way (eg: using tags to make a new menu item appear). Users must understand the specific tiddlywiki well, but not necessarily programming in general. In such an application adding a button or fixing a typo does not require a software developer’s participation.

The majority of text-based applications (including the original wiki functionality) become more valuable when collaboration is possible. TiddlyWiki has grown to become a fantastic tool as a personal wiki. Imagine the possibilities if we could make it a collaborative experience!

The road to collaboration

I believe a theoretical collaborative multi-user TiddlyWiki’s ideal users are small teams (less than a dozen individuals). Even in such a small, relatively high-trust scenario, certain guarantees are needed:

  1. Editing a tiddler should not ever result in accidentally overwriting another user’s simultaneous changes.
  2. It shouldn’t be possible for a user of the wiki to run un-sandboxed code in other users’ browsers (more on this in a bit).
  3. Administrators of the wiki should be able to add / remove users.
  4. There should be some form of access control (not everyone should be able to read / edit every tiddler).
  5. The complexity and cost of installing and operating the wiki should be modest.

It would be great if we could use an existing system for storing tiddlers. Unfortunately, I didn’t find anything that worked. The first point is difficult to achieve if the entire wiki is a single file. Solutions like the -otherwise excellent- TiddlyDrive https://tiddlydrive.github.io/ and TiddlyWiki in the Sky https://twcloud.github.io/tw5-dropbox/ thus don’t scale to the dozen-user teams I targeted. Storing each tiddler as a separate file in Google Drive or Dropbox would lead to very slow startup times and edit conflicts would be difficult to handle.

Collaborative web apps traditionally store their state in a transactional database to solve the problems around simultaneous user actions. Unfortunately installing and operating traditional web apps has been complex and time-consuming.

Firebase as a user-friendly backend

Installing and operating a database like postgres is not something a non-developer should have to do. Building on supabase as @boris mentioned is indeed an option, but at the time I started tw5-firebase, it wasn’t yet announced. Even today, it still lacks many of the components offered by firebase. Fission might be the perfect platform for a collaborative tiddlywiki server someday, but until then, we have Firebase.

Firebase attempts to provide solutions for all the traditional needs of a collaborative web apps while providing user-friendly dashboards. Namely:

  • Authentication (firebase auth integrates with over a dozen identity providers)
  • Static asset hosting (with integrated CDN)
  • Database (firestore)
  • User file hosting (different from static asset hosting, uses google storage)
  • Server-side functions (run server-side javascript without managing servers)

All this is bundled up and nicely integrated. The various components know about each other. Firebase is well documented, with plenty of examples.

Of course, there are disadvantages:

  • I’m just as disillusioned with Google as everyone else. The original “do no evil” mantra has seemingly disappeared - perhaps unavoidable at their scale, but still disappointing.
  • Cloud functions requires users to opt into billing by providing their credit card. It gave me a pause too, I’m sure a lot of people would hesitate to proceed at that point.
  • Firestore is not my favorite database. It’s impossible to run the real thing locally during development, performance is hard to analyze (no EXPLAIN). The backup / restore tools are pretty lame. The way indexes work is hacky. There’s a surprisingly low ceiling to document size. It’s not horrible, but it’s not great either.
  • Cloud Functions are the only supported way of executing custom server-side logic, which means that long-running processes with lots of in-memory state don’t work. This brings us to our next point:

Why not use the built-in node.js server?

TiddlyWiki5 comes with a built-in webserver. Projects like Bob https://github.com/OokTech/TW5-Bob build on it. It works really well locally. Unfortunately, as I mentioned at the end of the previous section, it’s architecture is not very firebase friendly. On startup, the TW5 node app reads all tiddlers. There is no further reading of tiddlers from the filesystem. Once loaded, the “source of truth” for tiddlers is the node process’ memory.

If we were to naively move this to Firebase (and swap the filesystem for firestore), serving every single request would require reading all tiddlers before it even got to processing the request itself. Also, firebase cloud functions run in parallel, which would lead to users accidentally overriding each others’ edits. So we need something else.

Note that this isn’t really a firebase-specific issue: collaborative web apps generally have a central ACID-compliant database storing application state, and -potentially several- stateless web server processes running the app’s code. The web server processes can (and in the case of Cloud Functions do) get replaced frequently; the contents of their memory should not need to survive multiple requests. AWS Lambda, supabase, etc would all require following the same architecture.

The built-in node.js server is built the way it is because it’s meant to be super-compact, treating node.js as just another execution environment. It’s brilliant but doesn’t scale. Since TiddlyWiki is designed to be a personal wiki, it doesn’t need to.

Current TW5-firebase architecture

The current architecture (what’s currently in master:

  1. A static HTML file containing a very minimal customized TiddlyWiki loads and authenticates the user using firebase auth.
  2. Once authenticated, tiddlers are read from firestore via HTTP request to a firebase function.
  3. Updating tiddlers also happens via HTTP requests to firebase functions (which then write to firestore).
  4. Importing non-text tiddlers uploads them to firestore storage. A tiddler with a _canonical_uri field points to the real URL of the uploaded file.

This is already useful, but it doesn’t do all that’s required for collaboration:

  • No user-friendly way to add users (although a CLI tool for this is included).
  • No user-friendly way to set permissions. Access control is managed by placing tiddlers in bags. Individual tiddlers don’t have rights. Moving tiddlers between bags should be easy for a user and atomic. They’re not.
  • Currently, only admin users can add / update code in the wiki to protect users against other (malicious / playful) users. The wiki has access to firebase credentials, however, so an admin could impersonate the user and delete tiddlers in another wiki they have access to.

Next Architecture

  • Firestore supports listening to changes in the DB. This means when one user makes an edit to a tiddler (and save the draft), any other user editing the same tiddler could be instantly modified. They would “merge” their versions of the tiddler before the second user saves, hopefully eliminating most “conflict on save” situations.
  • In order to isolate user code within the wiki from firebase credentials, the wiki itself should run within a sandboxed iframe, while the “outer” frame communicates with firebase.

Current Status

In short, TW5-firebase is a (mostly) working prototype. It shouldn’t eat your data, but it is definitely not read for collaboration.

The master branch of tw5-firebase should work. I plan on fixing the issues @stobot faced with installation. Please note that the optimistic locking logic is not perfect: it does occasionally detect conflicts when it shouldn’t. This is not handled well by the current UI (the “save error” dialog appears). They can be resolved by deleting the draft tiddlers in question, but it’s not pretty.

The tiddler-version-manager at https://github.com/neumark/tw5-firebase/tree/tiddler-version-manager branch is an unfinished attempt at introducing the changes I mentioned in the “next architecture” section. I hope to continue someday, but haven’t been able to work on it for the last couple of months.

PS: I’m a bit upset that I’m only allowed to post 2 links!

2 Likes
#27

I suspect that is because you are new to the forum. Hopefully one of the @moderators can bump up your level.

#28

@stobot You can not run bash scripts with PowerShell. bash is unix and PowerShell is windows. … That doesn’t work!

#29

WSL Windows Subsystem for Linux is a completely separated OS. If you installed nodejs npm and yarn on windows. WSL can not use it from the ubuntu shell.

The mount point is just a convenient way to access files only. So you can see files from ubuntu that are “hosted” in the windows file system.

If you open the ubuntu shell you need to install node and yarn in the ubuntu shell. Then run the scripts from there.

BUT there still will be some problems connecting the IP addresses, which I don’t know from hart atm.

#30

I did set your state to Basic User, so you should be allowed to use more than 2 links. I think the system will also send you some more info mails. pls read them.

1 Like
#31

Thanks @pmario - I think following @saqimtiaz’s advice (quoted below) helped from an install standpoint.

So the next thing to try is what @Mark_S and @saqimtiaz mentioned - re-doing the git clone bit to replace what I may have disturbed with adding echo commands (and I’ll try the echos again as @Mark_S suggested before I do that).

If you’re right and it is an IP address issue, would I have to more fully install a proper linux alongside Windows and boot into it to try it do you think? Unfortunately while that sounds interesting, it may be too far beyond my current capabilities to figure out, but I’ll cross that bridge when I get there. I won’t be able to try more until much later today - thanks all for your assistance so far.

#32

Oh yeah. So if you edit the scripts do it inside the linux environment, and maybe use a linux friendly editor. vi is usually installed. vi is the best editor in the universe, except for all the others.

But probably you have to all the shell steps (including installing git) under the linux shell.

#33

No … For testing I’d go with WSL since it has a much better connection between Windows file system and unix filesystem.

If you boot into a completely new OS you will open up a lot of different problems with filesharing between OSes.

#34

Did a short test with the default TW nodejs server. It’s actually pretty simple to run a node server in WSL ubuntu and connect to it from windows on the same machine.

In the ubuntu terminal just type

m@ubuntu:~$ ip -c -4 address

It should give you something similar to

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 172.31.01.02/20 brd 172.xx.yy.zz scope global eth0
       valid_lft forever preferred_lft forever

The important part here is the eth0 inet 172.31.01.02/20 address. It is the address you’ll need to start the tw node server

tiddlywiki myNewWiki/ --listen host=172.31.01.02

Then in the windows browser the TW server will be accessible with: http://172.31.01.02:8080/ …

That’s just a description how I did connect the default tw nodejs server from WSL ubuntu to Windows 10. The ip commands should also work for you but you’ll need to adjust the settings to the firebase setup, which I didn’t have a look at.

#35

Had a chance to start over this morning and try again not using Notepad for any of my editing.

Still didn’t work but got new error messages. Full transcript below in case that’s useful to @neumark when he has a chance to look at this.

Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.10.16.3-microsoft-standard-WSL2 x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Sat Nov 13 08:12:52 EST 2021

  System load:  0.0                Processes:             8
  Usage of /:   1.0% of 250.98GB   Users logged in:       0
  Memory usage: 3%                 IPv4 address for eth0: 172.31.168.25
  Swap usage:   0%


0 updates can be applied immediately.



This message is shown once a day. To disable it please create the
/home/stobbea/.hushlogin file.
stobbea@STOBBE-LAPTOP:/mnt/c/Users/stobb$ git clone https://github.com/neumark/tw5-firebase.git
Cloning into 'tw5-firebase'...
remote: Enumerating objects: 2811, done.
remote: Counting objects: 100% (2089/2089), done.
remote: Compressing objects: 100% (1313/1313), done.
remote: Total 2811 (delta 1159), reused 1572 (delta 667), pack-reused 722
Receiving objects: 100% (2811/2811), 6.77 MiB | 1.07 MiB/s, done.
Resolving deltas: 100% (1533/1533), done.
stobbea@STOBBE-LAPTOP:/mnt/c/Users/stobb$ cd tw5-firebase
stobbea@STOBBE-LAPTOP:/mnt/c/Users/stobb/tw5-firebase$ yarn install
yarn install v1.22.17
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning "firebase-admin > @firebase/database > @firebase/auth-interop-types@0.1.6" has unmet peer dependency "@firebase/app-types@0.x".
warning " > @istanbuljs/nyc-config-typescript@1.0.1" has unmet peer dependency "ts-node@*".
[4/4] Building fresh packages...
Done in 147.17s.
stobbea@STOBBE-LAPTOP:/mnt/c/Users/stobb/tw5-firebase$ yarn run firebase login:ci
yarn run v1.22.17
$ /mnt/c/Users/stobb/tw5-firebase/node_modules/.bin/firebase login:ci

Visit this URL on this device to log in:
https://accounts.google.com/o/oauth2/auth?client_id=563584335869-fgrhgmd47bqnekij5i8b5pr03ho849e6.apps.googleusercontent
.com&scope=email%20openid%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloudplatformprojects.readonly%20https%3A%2F%2Fwww
{
.googleapis.com%2Fauth%2Ffirebase%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform&response_type=code&state=10
{
36477684&redirect_uri=http%3A%2F%2Flocalhost%3A9005

Waiting for authentication...

✔  Success! Use this token to login on a CI server:

1//04H6U2j9oONGNCgYIARAAGAQSNwF-L9IrqN3nroK9BEb2eFjzWlqA-wTuv5gHlatEjKFlavguT7Ebro3WG-w6mum5BJ6lEkfz0_k

Example: firebase deploy --token "$FIREBASE_TOKEN"

Done in 25.36s.
stobbea@STOBBE-LAPTOP:/mnt/c/Users/stobb/tw5-firebase/etc$ vi keys.json
stobbea@STOBBE-LAPTOP:/mnt/c/Users/stobb/tw5-firebase/etc$ vi config.json
stobbea@STOBBE-LAPTOP:/mnt/c/Users/stobb/tw5-firebase/etc$ cd ..
stobbea@STOBBE-LAPTOP:/mnt/c/Users/stobb/tw5-firebase$ yarn run deploy
yarn run v1.22.17
$ ./scripts/build.sh && ./scripts/deploy.sh
Error: /mnt/c/Users/stobb/tw5-firebase/scripts/../etc/config.staging.json and /mnt/c/Users/stobb/tw5-firebase/scripts/../etc/keys.staging.json must exist
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

So… new error messages. Seems like config.staging.json and keys.staging.json are missing, though they don’t get generated in the initial git.clone, nor do they get reference in the build.sh script that I was looking at before…

#36

This is where that error comes from

You might brick things, but I would be tempted to copy the keys.json and config.json to keys.staging.json and config.staging.json and give it another go.

Edit: you may have similar issues with the service-account-key.json file:

#37

Thanks @saqimtiaz that (doing the copy) returns

stobbea@STOBBE-LAPTOP:/mnt/c/Users/stobb/tw5-firebase/etc$ cp keys.json keys.staging.json
stobbea@STOBBE-LAPTOP:/mnt/c/Users/stobb/tw5-firebase/etc$ cp config.json config.staging.json
stobbea@STOBBE-LAPTOP:/mnt/c/Users/stobb/tw5-firebase/etc$ cd ..
stobbea@STOBBE-LAPTOP:/mnt/c/Users/stobb/tw5-firebase$ yarn run deploy
yarn run v1.22.17
$ ./scripts/build.sh && ./scripts/deploy.sh
/mnt/c/Users/stobb/tw5-firebase /mnt/c/Users/stobb/tw5-firebase
/mnt/c/Users/stobb/tw5-firebase /mnt/c/Users/stobb/tw5-firebase
fs.js:114
    throw err;
    ^

Error: ENOENT: no such file or directory, open '/mnt/c/Users/stobb/tw5-firebase/generated/jsonschema/tiddlerdata.json'
    at Object.openSync (fs.js:443:3)
    at Object.writeFileSync (fs.js:1194:35)
    at Object.<anonymous> (/mnt/c/Users/stobb/tw5-firebase/build/schemabuilder.js:18:8)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
    at startup (internal/bootstrap/node.js:283:19)
/mnt/c/Users/stobb/tw5-firebase
/mnt/c/Users/stobb/tw5-firebase/scripts/build-prerequisites.sh: line 9: /mnt/c/Users/stobb/tw5-firebase/scripts/../generated/config/config.json: No such file or directory
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I appreciate all of the help, but I’m also becoming aware that even if I got it installed, I might not even know how to configure it at the end. The instructions given mention setting up users, but the instructions don’t cover any of it. I really don’t want to waste any of everybody’s precious time here, I should probably just be patient and wait for any of:

  • @neumark to have time to take a look / get a little further
  • @joshuafontany to get closer on his multi-user Yjs effort
  • @Charlie_Veniot who has a multi-user experiment that’s on the shelf
  • @inmysocks to come back to TiddlyWiki and update to 5.2.0 or go further

It feels like a few people are almost there :slight_smile:

1 Like
#38

@stobot … Please make sure, that you revoke your access tokens at firebase.

In one of your posts is a string, that looks like an access key. …

There also is a client_id … I don’t know if it is project specific and designed to be publicly available. … Please check the info at your account. If firebase says that you should keep it private, you probably need to reset the project … or even the account.

You should never ever post user IDs and access keys publicly!

#39

Thanks @Mario - I deleted the “project” - hopefully that was sufficient. I appreciate you taking the time to protect me from myself here :slight_smile:

#40

Hi guys!

Thanks for the patience, @stobot!
I used TW5-firebase for months, dogfooding my own solution. Recently I’ve replaced it with TiddlyDesktop with Syncthing to sync the wiki folder to other machines. For what I’m doing, this is currently a better solution. It may work for you as well (although you did mention you’re not admin on the machines, but maybe syncthing has a portable edition you can run regardless).

I plan on updating the tw5-firebase documentation to be more windows user friendly. I also realized that it’s possible to make tw5-firebase work without cloud functions, which would eliminate the need to provide credit card details to firebase when signing up.

Hope you find a good solution for your team!
Peter

1 Like
#41

I for one will look forward to this, thanks.