Telegram bot for tiddlywiki

#1

Screenshot_20220524-180412_Twitter
Screenshot_20220524-180412_Twitter1054×564 85.9 KB

Saw this old post from 2016 regarding telegram bot for tiddlywiki from twitter handle of tiddlywiki

Anyone has used it . The attached link doesn’t seem to work.

#2

Is that bot active in Telegram? Looks way dead.

If I had the time I’d certainly write more on Telegram about TW.

It is an extremely good approach to discussion.
But, unless you are on it, it is very obscure.

TT

#3

I have not used this before, but having recently switched from Logseq, the ability to jot down or even speak into telegram to jot down a quick note that ends up in your PKMS was something I missed

I was using Lupin (Lupin/readme.md at master · akhater/Lupin · GitHub) for Logseq, which I had setup on a docker.
This bot takes your message and adds the note to the daily journal file stored on GitHub

Since I was already using the TidGi app to store tiddlers on GitHub, I played with Lupin’s config as well as the tiddlyfile.info configs and sure enough, was easy as to make the note from Logseq appear straight into tiddlywiki as an external note. A small config change would also make it trivial to store directly into TW itself, but I have left it like this for the moment just in case I jump back to Logseq.

Happy to share additional details if such an approach works for you

5 Likes
#4

For users who can cope with technical stuff I think that note is very useful!!

TT

#5

Thanks for your response.
I haven’t used TW on node js yet. If you could share more details, I will try it out.

#6

Sure, am not very good with writing instructions, but will give it a shot. I am not sure some of the TW specific stuff is the best way to achieve this - but perhaps some of the long term users could comment if something is not quite optimal or perhaps leave suggestions.

So what we are trying to achieve here is just to use share content from the mobile through the Telegram app - bookmarks, ideas on a bus trip to the office, etc. Notes land in an “Inbox” which sits just outside of the tiddlers folder, and then get sucked in as external content, which can be edited / saved as normal tiddlers.

So first step is to install Tiddlywiki on Node. You can follow instructions here.
Another option might be TiddlyServer or yet another option is Tidgi. You can
follow any of these - main point being there has to be a sync with github, which the last option makes very easy.

Next, install Lupin as per instructions - easiest option is to have it on docker ; should be fairly straight forward.

Next update the Lupin config file. Most entries can be left as is. My personal setup involves logseq, so I have tweaked the below as it might look like if you were using TW by itself.

Key config entries:


; LogSeq GitHub repo - whatever yours is called
GitHubRepo = Stroll

; format of your journal files - Should be same format as how your TW names journals
journalsFilesFormat = %Y_%m_%d

; journal files extension .md / .org - I choose .md as its far more interchangeable. Note 
; that if you want .tid or something, you will also need to change the source code, which
; I have tried to avoid
journalsFilesExtension = .md

; change this value if you want your mobile Journal to reside in a different folder
journalsFolder = mobile

Finally, some config changes on your tiddlywiki:

  1. Create a folder directly under your base tiddlywiki folder called ‘mobile’

  2. Create a folder called Inbox under ‘tiddlers’ folder. Under this folder, create a file called
    tiddlywiki.files

  3. Your folder structure should look something like:
    /
    |---- tiddlers/Inbox/tiddlywiki.files
    |---- mobile

Copy the below into the tiddlywiki.files file:

{
    "directories": [
        {
            "path": "../../mobile",
            "filesRegExp": "^.*\\.md$",
            "isTiddlerFile": false,
            "isEditableFile": true,
            "fields": {
                "title": {"source": "basename-uri-decoded"},
                "created": {"source": "created"},
                "modified": {"source": "modified"},
                "type": "text/x-markdown",
                "tags": ["telegram", "externalnote", "Inbox"]
            }
        }
  ]
}

This should be it. Now if you send a message in Telegram, it will add a timestamped
entry into the file mobile/Inbox. The tiddlywiki.files entry will suck this up and show it as external content in your TW. You can then literally treat it as an Inbox and move the
message into other tiddlers as you see fit, or just leave it there. Any changes you make there will write back to the mobile/Inbox file.

Just remember to always sync with GitHub.

Hope you find it useful.

4 Likes
#7

Thanks for sharing the details @Himanshu_Shukla :+1:

I will try it out and give you feedback.

#8

I’m planning to try this out myself soon. @arunnbabu81, as someone who’s done this, do you have any advice or things you wish you’d known before trying it?

#9

Sorry I was not able to try it out. I had some issues with docker installation I guess. It was sometime back, so i don’t remember correctly.

#10

Thank you for the quick reply! And no problem. I saw it was a while ago and didn’t expect detailed memory. (:slight_smile:

#11

hello! to be able to have quick notes from mobile to Tiddlywiki is something that interests me very much! If there are more people onboard and/or we can arrange for a good plan for implementation, I might be able to tackle this — I have experience with telegram bots, and it shouldn’t be too hard. I just wouldn’t like to implement something that is only useful for myself :grin:

3 Likes
#12

This would be fantastic! Let me know how you’d like to do this— I’m happy to do as much collaboration as you’d like but I also get it if you’d like to take a stab at it by yourself first.

In any case, here’s the reply I got from another thread discussing a more specific version of this (with GitHub pages) Telegram edits with Github saver (and Github pages) - #14 by David_Beijinho. So that might be helpful in this project

Telegram edits with Github saver (and Github pages)
#13

I’ll keep this in mind, and I’ve had it for the last few days. Here are some thoughts:

I think the biggest “technical problem” is that TiddlyWiki has so many different ways of being deployed, it is difficult for a general solution to fit all the cases. For example, I have automated the creation of tiddlers for a personal project of mine, but since I am running a node, and have access to the underlying system in which my TiddlyWiki instance (henceforth, TWi) runs, I had lots of liberty to experiment as I wanted.

Not assuming direct connectivity to the TWi or system access (suppose, for example, the limited environment of TiddlyHost), I think it’s necessary to have a sort of middle-man to which the TWi can connect and retrieve content from (otherwise, one could just “bruteforce” and create the .tid files directly on their directory). From what I understand of the above solution (granted, I read it only in passing, maybe tomorrow I’ll be more thorough), that would be the role of a Github Pages instance (GPi), or other alternative (ideally a free service, of course, but it could also be something that one would run on their own Raspberry Pi, for example)

Then, the workflow would be something like

  1. user messages their own Telegram bot;

  2. telegram bot routes the information to GPi, which has some sort of script permanently listening for inputs;

  3. whenever TWi is connected to the internet (or at regular intervals of X minutes) it pings the GPi to retrieve tiddlers — this would be done via some javascript plugin running on the TWi browser; IIRC, Javascript is disabled by default (and generally discouraged), but it would essentially be a plugin to make GET requests to the telegram bot.

I think that is pretty much it? It is definitely not complicated, but there are a lot of moving parts and I’d rather make my thought process public to catch any problems sooner rather than later.

@noa, from what I read on another comment of yours about this topic, it seems like you are starting to do some code? In that case, for sure I will not tackle this own my own — the more the merrier! :smiley: let’s keep the discussion going and see how it goes.

(I also saw a bit of @David_Beijinho’s solution, but, again, only in passing — it’s late here! — so tomorrow I’ll look deeper into it.)

1 Like
#14

This sounds exactly right!

I haven’t actually gotten around to starting yet. And when I’m programming this kind of connection to multiple systems, I don’t yet have the intuitions for what tools to use or best structures for the code. So I’m excited to learn and this is something I care a lot about getting better at, but I just wanted to give you the heads-up about what to expect. I’m probably not going to be the most helpful collaborator in the world :slight_smile:

#15

I think at this very moment the most difficult step is the first one — getting started! The rest will come naturally :grin: being motivated and caring about things is, in my experience, a decisive factor; programming is as much about getting things right as it is to getting them done — of course, ideally without much frustration, overwhelming doses of caffeine ,sleepless nights, etc. etc.

I was looking into your other post in which you detail your setup; it seems like you use GitHub Saver? From what I understand, yout wiki lives essentially in a git repository; you connect to it via the browser, correct? Is there any login, authentication, etc? How exactly do you make changes to the wiki?

A good starting point, and probably a generally good practice in programming, is to create a “sandbox” testing environment; in this case, something that would reproduce your current setup as closely as possible, and we build it up from there.

And I forgot to mention something: the Telegram bot needs to be running 24/7 (if you want to be able to send notes at any given time); a few years ago, one would use something like Heroku, because they had some really good plans — today, I’m not sure what is out there (I have also used stuff like DigitalOcean, it was not too expensive). So that is another thing to figure out…

Does that make sense? I re-read this frequently to make sure things are clear, but it’s a bit late here and I might be either overlooking very basic things or complicating too much. :grinning:

#16

:slight_smile:Thank you so much!

That’s right. The login is a saved password within the browser (in the tiddlywiki “save” options). So to make changes, I just click the “save” button within tiddlywiki and it pushes the whole HTML to github. Now, in that thread, I was told (and this sounds right to me) that it might make more sense both for a telegram bot and for such a large file to do something instead that saves individual .tid files and then uses github actions to make the larger HTML out of those separate saved .tids. I’m still looking into how to do this since I (probably overly-optimistically) feel like I understand the basics of how to program the rest. Does anyone have any examples for me to look at for this kind of saving?

I have no idea if this is best practice, but two years ago, I created a telegram to google sheets bot using a google cloud VM which has been running 24/7 ever since and ended up costing less than a dollar over that entire period. So I’d lean towards doing that again for this project unless anyone has another suggestion.

Thank you again for all your patient thoughtfulness! I’ll get back to you when I’ve done the next steps (which might be in a bit due to some extra work this week).

#17

So I’ve created a demo tiddlywiki which is saving to github and accessed through github pages and I set up a telegram bot. But when I tried to have the bot add a line to an existing tiddler within index.html, it messed up the entire index.html. As far as I can tell, the main thing that happened was removing all ascii “\u00”…

So I created a very simple local version of the python script which just takes a basic index.html tiddlywiki (where I manually added a tiddler called “ATiddlerToEdit”) saved in the same directory and should add “IADDEDTHISTEXT” to the tiddler, “ATiddlerToEdit”. The same problem is still happening. Does anyone know what’s going wrong in the json processing?

import re
import json

def modify_text_in_html(content, title, new_text):
    """
    This function looks for the JSON object in the <script> tag in the HTML content,
    searches for the `title`, and appends `new_text` to the `text` field of that object.
    """

    # Define the pattern to locate the <script> tag containing JSON
    script_pattern = re.compile(r'(<script class="tiddlywiki-tiddler-store" type="application/json">)(.*?)(</script>)', re.DOTALL)
    
    # Search for the script tag that contains the JSON array
    match = script_pattern.search(content)
    
    if not match:
        return None  # Return None if no matching script tag is found

    # Extract the JSON string from within the <script> tag
    json_str = match.group(2)

    # Load the JSON content (assuming it's an array of objects)
    try:
        tiddlers = json.loads(json_str)
    except json.JSONDecodeError as e:
        print(f"Error decoding JSON: {e}")
        return None

    # Find the tiddler with the matching title
    modified = False
    for tiddler in tiddlers:
        if tiddler.get("title") == title:
            # Append the new text to the "text" field
            tiddler["text"] += f" {new_text}"
            modified = True
            break

    if not modified:
        return None  # Return None if no matching title is found

    # Convert the modified tiddlers back to a JSON string
    updated_json_str = json.dumps(tiddlers, indent=2, ensure_ascii=False)

    # Reinsert the modified JSON back into the <script> tag
    modified_content = content.replace(json_str, updated_json_str)

    return modified_content


file_path = "index.html"


with open("index.html",'r', encoding='utf-8') as tiddly_text:
    print("hello")
    readable = tiddly_text.read() 
    print(readable[0:100])
    #modify_text_in_html(readable, "ATiddlerToEdit", "IADDEDTHISTEXT")

    #below is just for the local version
    # Modify the content by adding new text to the tiddler
    modified_content = modify_text_in_html(readable, "ATiddlerToEdit", "IADDEDTHISTEXT")

    if modified_content:
        # If the content was successfully modified, write it back to the file
        with open(file_path, 'w', encoding='utf-8') as tiddly_text:
            tiddly_text.write(modified_content)
        print("Modification successful.")
    else:
        print("Modification failed or no matching title found.")
#18

Encoding:

By default, all tiddlers are now stored as an array of JSON objects inside a <script> tag with the class attribute “tiddlywiki-tiddler-store” and the type “application/json”. For better readability, every tiddler object begins on a new line. There are no longer any restrictions on characters that are allowed in tiddler field names. However, <script> tag store areas must encode the < character to \u003c

The easiest option would be to prepend an extra script tag to the TiddlyWiki file, see the end of this tiddler under the heading “Multiple store areas and precedence”: https://tiddlywiki.com/dev/#Data%20Storage%20in%20Single%20File%20TiddlyWiki

#19

Thank so much! I might be misunderstanding, but I’d like to be able to both add new tiddlers (which I think is what you’re suggesting) and make edits to existing tiddlers. Is there a way to edit my code to have the right < → \u003c encoding so that I can also make edits?

#20

try this after the line that converts the updated tiddler back into JSON:

# Convert the modified tiddlers back to a JSON string
updated_json_str = json.dumps(tiddlers, indent=2, ensure_ascii=False)
updated_json_str = updated_json_str.replace("<", "\\u003C")

Please note that I have not worked through the rest of the python code and am assuming that is correctly finding the JSON store and replacing it with the updated store.

If that fails, suggest performing a diff of the original file and the modified file to understand what other changes are being introduced. Use an empty TiddlyWiki containing just a single tiddler to make the diff easier to read.

1 Like