Thanks for sharing the details @Himanshu_Shukla 
I will try it out and give you feedback.
I’m planning to try this out myself soon. @arunnbabu81, as someone who’s done this, do you have any advice or things you wish you’d known before trying it?
Sorry I was not able to try it out. I had some issues with docker installation I guess. It was sometime back, so i don’t remember correctly.
Thank you for the quick reply! And no problem. I saw it was a while ago and didn’t expect detailed memory. (
hello! to be able to have quick notes from mobile to Tiddlywiki is something that interests me very much! If there are more people onboard and/or we can arrange for a good plan for implementation, I might be able to tackle this — I have experience with telegram bots, and it shouldn’t be too hard. I just wouldn’t like to implement something that is only useful for myself 
3 Likes
This would be fantastic! Let me know how you’d like to do this— I’m happy to do as much collaboration as you’d like but I also get it if you’d like to take a stab at it by yourself first.
In any case, here’s the reply I got from another thread discussing a more specific version of this (with GitHub pages) Telegram edits with Github saver (and Github pages) - #14 by David_Beijinho. So that might be helpful in this project
I’ll keep this in mind, and I’ve had it for the last few days. Here are some thoughts:
I think the biggest “technical problem” is that TiddlyWiki has so many different ways of being deployed, it is difficult for a general solution to fit all the cases. For example, I have automated the creation of tiddlers for a personal project of mine, but since I am running a node, and have access to the underlying system in which my TiddlyWiki instance (henceforth, TWi) runs, I had lots of liberty to experiment as I wanted.
Not assuming direct connectivity to the TWi or system access (suppose, for example, the limited environment of TiddlyHost), I think it’s necessary to have a sort of middle-man to which the TWi can connect and retrieve content from (otherwise, one could just “bruteforce” and create the .tid files directly on their directory). From what I understand of the above solution (granted, I read it only in passing, maybe tomorrow I’ll be more thorough), that would be the role of a Github Pages instance (GPi), or other alternative (ideally a free service, of course, but it could also be something that one would run on their own Raspberry Pi, for example)
Then, the workflow would be something like
-
user messages their own Telegram bot;
-
telegram bot routes the information to GPi, which has some sort of script permanently listening for inputs;
-
whenever TWi is connected to the internet (or at regular intervals of X minutes) it pings the GPi to retrieve tiddlers — this would be done via some javascript plugin running on the TWi browser; IIRC, Javascript is disabled by default (and generally discouraged), but it would essentially be a plugin to make GET requests to the telegram bot.
I think that is pretty much it? It is definitely not complicated, but there are a lot of moving parts and I’d rather make my thought process public to catch any problems sooner rather than later.
@noa, from what I read on another comment of yours about this topic, it seems like you are starting to do some code? In that case, for sure I will not tackle this own my own — the more the merrier! 
let’s keep the discussion going and see how it goes.
(I also saw a bit of @David_Beijinho’s solution, but, again, only in passing — it’s late here! — so tomorrow I’ll look deeper into it.)
1 Like
This sounds exactly right!
I haven’t actually gotten around to starting yet. And when I’m programming this kind of connection to multiple systems, I don’t yet have the intuitions for what tools to use or best structures for the code. So I’m excited to learn and this is something I care a lot about getting better at, but I just wanted to give you the heads-up about what to expect. I’m probably not going to be the most helpful collaborator in the world
I think at this very moment the most difficult step is the first one — getting started! The rest will come naturally being motivated and caring about things is, in my experience, a decisive factor; programming is as much about getting things right as it is to getting them done — of course, ideally without much frustration, overwhelming doses of caffeine ,sleepless nights, etc. etc.
I was looking into your other post in which you detail your setup; it seems like you use GitHub Saver? From what I understand, yout wiki lives essentially in a git repository; you connect to it via the browser, correct? Is there any login, authentication, etc? How exactly do you make changes to the wiki?
A good starting point, and probably a generally good practice in programming, is to create a “sandbox” testing environment; in this case, something that would reproduce your current setup as closely as possible, and we build it up from there.
And I forgot to mention something: the Telegram bot needs to be running 24/7 (if you want to be able to send notes at any given time); a few years ago, one would use something like Heroku, because they had some really good plans — today, I’m not sure what is out there (I have also used stuff like DigitalOcean, it was not too expensive). So that is another thing to figure out…
Does that make sense? I re-read this frequently to make sure things are clear, but it’s a bit late here and I might be either overlooking very basic things or complicating too much. 
Thank you so much!
That’s right. The login is a saved password within the browser (in the tiddlywiki “save” options). So to make changes, I just click the “save” button within tiddlywiki and it pushes the whole HTML to github. Now, in that thread, I was told (and this sounds right to me) that it might make more sense both for a telegram bot and for such a large file to do something instead that saves individual .tid files and then uses github actions to make the larger HTML out of those separate saved .tids. I’m still looking into how to do this since I (probably overly-optimistically) feel like I understand the basics of how to program the rest. Does anyone have any examples for me to look at for this kind of saving?
I have no idea if this is best practice, but two years ago, I created a telegram to google sheets bot using a google cloud VM which has been running 24/7 ever since and ended up costing less than a dollar over that entire period. So I’d lean towards doing that again for this project unless anyone has another suggestion.
Thank you again for all your patient thoughtfulness! I’ll get back to you when I’ve done the next steps (which might be in a bit due to some extra work this week).
So I’ve created a demo tiddlywiki which is saving to github and accessed through github pages and I set up a telegram bot. But when I tried to have the bot add a line to an existing tiddler within index.html, it messed up the entire index.html. As far as I can tell, the main thing that happened was removing all ascii “\u00”…
So I created a very simple local version of the python script which just takes a basic index.html tiddlywiki (where I manually added a tiddler called “ATiddlerToEdit”) saved in the same directory and should add “IADDEDTHISTEXT” to the tiddler, “ATiddlerToEdit”. The same problem is still happening. Does anyone know what’s going wrong in the json processing?
import re
import json
def modify_text_in_html(content, title, new_text):
"""
This function looks for the JSON object in the <script> tag in the HTML content,
searches for the `title`, and appends `new_text` to the `text` field of that object.
"""
# Define the pattern to locate the <script> tag containing JSON
script_pattern = re.compile(r'(<script class="tiddlywiki-tiddler-store" type="application/json">)(.*?)(</script>)', re.DOTALL)
# Search for the script tag that contains the JSON array
match = script_pattern.search(content)
if not match:
return None # Return None if no matching script tag is found
# Extract the JSON string from within the <script> tag
json_str = match.group(2)
# Load the JSON content (assuming it's an array of objects)
try:
tiddlers = json.loads(json_str)
except json.JSONDecodeError as e:
print(f"Error decoding JSON: {e}")
return None
# Find the tiddler with the matching title
modified = False
for tiddler in tiddlers:
if tiddler.get("title") == title:
# Append the new text to the "text" field
tiddler["text"] += f" {new_text}"
modified = True
break
if not modified:
return None # Return None if no matching title is found
# Convert the modified tiddlers back to a JSON string
updated_json_str = json.dumps(tiddlers, indent=2, ensure_ascii=False)
# Reinsert the modified JSON back into the <script> tag
modified_content = content.replace(json_str, updated_json_str)
return modified_content
file_path = "index.html"
with open("index.html",'r', encoding='utf-8') as tiddly_text:
print("hello")
readable = tiddly_text.read()
print(readable[0:100])
#modify_text_in_html(readable, "ATiddlerToEdit", "IADDEDTHISTEXT")
#below is just for the local version
# Modify the content by adding new text to the tiddler
modified_content = modify_text_in_html(readable, "ATiddlerToEdit", "IADDEDTHISTEXT")
if modified_content:
# If the content was successfully modified, write it back to the file
with open(file_path, 'w', encoding='utf-8') as tiddly_text:
tiddly_text.write(modified_content)
print("Modification successful.")
else:
print("Modification failed or no matching title found.")
Encoding:
By default, all tiddlers are now stored as an array of JSON objects inside a
<script>tag with theclassattribute “tiddlywiki-tiddler-store” and thetype“application/json”. For better readability, every tiddler object begins on a new line. There are no longer any restrictions on characters that are allowed in tiddler field names. However,<script>tag store areas must encode the<character to\u003c
The easiest option would be to prepend an extra script tag to the TiddlyWiki file, see the end of this tiddler under the heading “Multiple store areas and precedence”: https://tiddlywiki.com/dev/#Data%20Storage%20in%20Single%20File%20TiddlyWiki
Thank so much! I might be misunderstanding, but I’d like to be able to both add new tiddlers (which I think is what you’re suggesting) and make edits to existing tiddlers. Is there a way to edit my code to have the right < → \u003c encoding so that I can also make edits?
try this after the line that converts the updated tiddler back into JSON:
# Convert the modified tiddlers back to a JSON string
updated_json_str = json.dumps(tiddlers, indent=2, ensure_ascii=False)
updated_json_str = updated_json_str.replace("<", "\\u003C")
Please note that I have not worked through the rest of the python code and am assuming that is correctly finding the JSON store and replacing it with the updated store.
If that fails, suggest performing a diff of the original file and the modified file to understand what other changes are being introduced. Use an empty TiddlyWiki containing just a single tiddler to make the diff easier to read.
1 Like
Ah ha! I hadn’t realized it was this simple. Thank you – this works!
1 Like
So in case this is helpful for anyone, here’s the code that allows me to telegram my bot something like /editfile ATiddlerToEdit ADDED_TEXT! and it will add “ADDED_TEXT!” to ATiddlerToEdit. The extra setup required is to create your telegram bot and create a .env file in the same directory with a telegram token for your bot and a github token for the repo where you’re storing the tiddlywiki.
My plan is to change this so I have specific bot commands for adding text to specific tiddlers. But I thought I’d upload the code at this stage in case anyone wants to make their own bot with something more like this workflow. Feel free to let me know what looks particularly clunky in this code – I’m always hoping to learn to be a better programmer!
import os
import requests
import re
import json
from dotenv import load_dotenv
import base64
from telegram import Update
from telegram.ext import Application, CommandHandler, CallbackContext
load_dotenv(dotenv_path="credentials.env")
# Loading environment variables
TELEGRAM_TOKEN = os.getenv('TELEGRAM_TOKEN')
GITHUB_TOKEN = os.getenv('GITHUB_TOKEN')
REPO_OWNER = os.getenv('REPO_OWNER')
REPO_NAME = os.getenv('REPO_NAME')
FILE_PATH = os.getenv('FILE_PATH')
# Helper function to modify HTML content
def modify_text_in_html(content, title, new_text):
"""
This function looks for the JSON object in the <script> tag in the HTML content,
searches for the `title`, and appends `new_text` to the `text` field of that object.
"""
# Define the pattern to locate the <script> tag containing JSON
script_pattern = re.compile(r'(<script class="tiddlywiki-tiddler-store" type="application/json">)(.*?)(</script>)', re.DOTALL)
# Search for the script tag that contains the JSON array
match = script_pattern.search(content)
if not match:
return None # Return None if no matching script tag is found
# Extract the JSON string from within the <script> tag
json_str = match.group(2)
# Load the JSON content
try:
tiddlers = json.loads(json_str)
except json.JSONDecodeError as e:
print(f"Error decoding JSON: {e}")
return None
# Find the tiddler with the matching title
modified = False
for tiddler in tiddlers:
if tiddler.get("title") == title:
# Append the new text to the "text" field
tiddler["text"] += f" {new_text}"
modified = True
break
if not modified:
return None # Return None if no matching title is found
# Convert the modified tiddlers back to a JSON string
updated_json_str = json.dumps(tiddlers, indent=2)
updated_json_str = updated_json_str.replace("<", "\\u003C")
# Reinsert the modified JSON back into the <script> tag
modified_content = content.replace(json_str, updated_json_str)
return modified_content
async def edit_file(update: Update, context: CallbackContext) -> None:
try:
# Extract the title and new text from user input
if len(context.args) < 2:
await update.message.reply_text('Usage: /editfile <title> <new_text>')
return
title = context.args[0]
new_text = ' '.join(context.args[1:])
# GitHub API request to get the file content metadata
file_url = f'https://api.github.com/repos/{REPO_OWNER}/{REPO_NAME}/contents/{FILE_PATH}'
headers = {
'Authorization': f'token {GITHUB_TOKEN}',
'Accept': 'application/vnd.github.v3+json'
}
response = requests.get(file_url, headers=headers)
# Check for errors in the response
if response.status_code != 200:
print(f"Error {response.status_code}: {response.text}")
await update.message.reply_text(f"Failed to fetch the file metadata: {response.text}")
return
file_data = response.json()
# File is too large; we need to fetch it via the Blob API using the git_url
git_url = file_data.get('git_url')
if not git_url:
await update.message.reply_text("Unable to retrieve git_url for the file.")
return
# Fetch the content from the GitHub Blob API
blob_response = requests.get(git_url, headers=headers)
# Check if the blob API call is successful
if blob_response.status_code != 200:
await update.message.reply_text(f"Failed to fetch the file blob: {blob_response.text}")
return
blob_data = blob_response.json()
file_content_base64 = blob_data.get('content')
# Decode the base64 content (HTML content in this case)
file_content = base64.b64decode(file_content_base64).decode('utf-8')
# Modify the HTML content
updated_content = modify_text_in_html(file_content, title, new_text)
if updated_content is None:
await update.message.reply_text(f'Title "{title}" not found in the file.')
return
# Encode the updated content back to Base64
encoded_content = base64.b64encode(updated_content.encode('utf-8')).decode('utf-8')
# Prepare the data to update the file on GitHub
sha = file_data['sha'] # Get the current SHA of the file
data = {
'message': f'Updated HTML content for title "{title}" via Telegram bot',
'content': encoded_content,
'sha': sha # Include the file's SHA to make the update
}
# GitHub API request to update the file
update_response = requests.put(file_url, headers=headers, data=json.dumps(data, ensure_ascii=True))
if update_response.status_code == 200:
await update.message.reply_text(f'HTML content for "{title}" updated successfully!')
else:
await update.message.reply_text(f'Failed to update the file: {update_response.json().get("message")}')
except Exception as e:
await update.message.reply_text(f'An error occurred: {str(e)}')
def main() -> None:
# Initialize the bot application
application = Application.builder().token(TELEGRAM_TOKEN).build()
# Add a command handler
application.add_handler(CommandHandler('editfile', edit_file))
# Start the bot
application.run_polling()
if __name__ == '__main__':
main()
1 Like
And here’s the version where I telegram “b hello world” and it will add “hello world” to the tiddler “b_edit”. In this version of the code, I’ve hardcoded in “a” to edit “a_edit” and “b” to edit “b_edit”.
import os
import requests
import re
import json
from dotenv import load_dotenv
import base64
from telegram import Update
from telegram.ext import Application, ContextTypes, filters, MessageHandler, CommandHandler, CallbackContext
load_dotenv(dotenv_path="credentials.env")
# Loading environment variables
TELEGRAM_TOKEN = os.getenv('TELEGRAM_TOKEN')
GITHUB_TOKEN = os.getenv('GITHUB_TOKEN')
REPO_OWNER = os.getenv('REPO_OWNER')
REPO_NAME = os.getenv('REPO_NAME')
FILE_PATH = os.getenv('FILE_PATH')
# Helper function to modify HTML content
def modify_text_in_html(content, title, new_text):
"""
This function looks for the JSON object in the <script> tag in the HTML content,
searches for the `title`, and appends `new_text` to the `text` field of that object.
"""
# Define the pattern to locate the <script> tag containing JSON
script_pattern = re.compile(r'(<script class="tiddlywiki-tiddler-store" type="application/json">)(.*?)(</script>)', re.DOTALL)
# Search for the script tag that contains the JSON array
match = script_pattern.search(content)
if not match:
return None # Return None if no matching script tag is found
# Extract the JSON string from within the <script> tag
json_str = match.group(2)
# Load the JSON content
try:
tiddlers = json.loads(json_str)
except json.JSONDecodeError as e:
print(f"Error decoding JSON: {e}")
return None
# Find the tiddler with the matching title
modified = False
for tiddler in tiddlers:
if tiddler.get("title") == title:
# Append the new text to the "text" field
tiddler["text"] += f" {new_text}"
modified = True
break
if not modified:
return None # Return None if no matching title is found
# Convert the modified tiddlers back to a JSON string
updated_json_str = json.dumps(tiddlers, indent=2)
updated_json_str = updated_json_str.replace("<", "\\u003C")
# Reinsert the modified JSON back into the <script> tag
modified_content = content.replace(json_str, updated_json_str)
return modified_content
async def edit_file(update, title, new_text) -> None:
try:
# GitHub API request to get the file content metadata
file_url = f'https://api.github.com/repos/{REPO_OWNER}/{REPO_NAME}/contents/{FILE_PATH}'
headers = {
'Authorization': f'token {GITHUB_TOKEN}',
'Accept': 'application/vnd.github.v3+json'
}
response = requests.get(file_url, headers=headers)
# Check for errors in the response
if response.status_code != 200:
print(f"Error {response.status_code}: {response.text}")
await update.message.reply_text(f"Failed to fetch the file metadata: {response.text}")
return
file_data = response.json()
# File is too large; we need to fetch it via the Blob API using the git_url
git_url = file_data.get('git_url')
if not git_url:
await update.message.reply_text("Unable to retrieve git_url for the file.")
return
# Fetch the content from the GitHub Blob API
blob_response = requests.get(git_url, headers=headers)
# Check if the blob API call is successful
if blob_response.status_code != 200:
await update.message.reply_text(f"Failed to fetch the file blob: {blob_response.text}")
return
blob_data = blob_response.json()
file_content_base64 = blob_data.get('content')
# Decode the base64 content (HTML content in this case)
file_content = base64.b64decode(file_content_base64).decode('utf-8')
# Modify the HTML content
updated_content = modify_text_in_html(file_content, title, new_text)
if updated_content is None:
await update.message.reply_text(f'Title "{title}" not found in the file.')
return
# Encode the updated content back to Base64
encoded_content = base64.b64encode(updated_content.encode('utf-8')).decode('utf-8')
# Prepare the data to update the file on GitHub
sha = file_data['sha'] # Get the current SHA of the file
data = {
'message': f'Updated HTML content for title "{title}" via Telegram bot',
'content': encoded_content,
'sha': sha # Include the file's SHA to make the update
}
# GitHub API request to update the file
update_response = requests.put(file_url, headers=headers, data=json.dumps(data, ensure_ascii=True))
if update_response.status_code == 200:
await update.message.reply_text(f'HTML content for "{title}" updated successfully!')
else:
await update.message.reply_text(f'Failed to update the file: {update_response.json().get("message")}')
except Exception as e:
await update.message.reply_text(f'An error occurred: {str(e)}')
async def edit_a_edit(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
title = "a_edit"
new_text_with_a = update.message.text
new_text = new_text_with_a[2:]
await edit_file(update, title, new_text)
async def edit_a_edit(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
title = "b_edit"
new_text_with_b = update.message.text
new_text = new_text_with_b[2:]
await edit_file(update, title, new_text)
def main() -> None:
# Initialize the bot application
application = Application.builder().token(TELEGRAM_TOKEN).build()
# Add a command handler
application.add_handler(MessageHandler(filters.Regex('(?i)^a .*'), edit_a_edit))
application.add_handler(MessageHandler(filters.Regex('(?i)^b .*'), edit_a_edit))
# Start the bot
application.run_polling()
if __name__ == '__main__':
main()
I checked wayback machine as a hailmary to see if any docs or files exist for this and no dice. I think this gem is lost to time sadly
1 Like
And here’s a version of the code that works with a tiddlywiki index.html saved on google drive. All you have to do is create a google cloud service account with access to your drive, share the index.html with it, and download the credentials to credentials.json stored in the same directory.
import logging
from telegram import Update
from telegram.ext import Application, ContextTypes, filters, MessageHandler, CommandHandler
from google.oauth2.service_account import Credentials
from googleapiclient.discovery import build
from googleapiclient.http import MediaIoBaseDownload, MediaFileUpload
import io
import os
import json
import re
from dotenv import load_dotenv
load_dotenv(dotenv_path="credentials.env")
SCOPES = ['https://www.googleapis.com/auth/drive']
TELEGRAM_TOKEN = os.getenv('TELEGRAM_TOKEN')
FILE_ID = os.getenv('FILE_ID')
def get_drive_service():
# Load the service account credentials from the JSON key file
creds = Credentials.from_service_account_file('credentials.json', scopes=SCOPES)
service = build('drive', 'v3', credentials=creds)
return service
async def get_file_content(file_id):
service = get_drive_service()
request = service.files().get_media(fileId=file_id)
fh = io.BytesIO() # File-like object for reading in memory
downloader = MediaIoBaseDownload(fh, request)
done = False
while not done:
status, done = downloader.next_chunk()
# After download, get the content
fh.seek(0)
content = fh.read().decode('utf-8') # Assuming the file is a text file
return content # Return the entire content
async def get_file(update: Update):
try:
file_id = FILE_ID # Get the file ID from the command
content = await get_file_content(file_id)
await update.message.reply_text(f"First 100 characters: {content[:100]}")
return content
except IndexError:
await update.message.reply_text("Please provide a file ID.")
except Exception as e:
await update.message.reply_text(f"Error: {e}")
async def modify_text_in_html(update, title, new_text):
"""
This function looks for the JSON object in the <script> tag in the HTML content,
searches for the `title`, and appends `new_text` to the `text` field of that object.
"""
content = await get_file_content(FILE_ID) # Await the coroutine
# Define the pattern to locate the <script> tag containing JSON
script_pattern = re.compile(r'(<script class="tiddlywiki-tiddler-store" type="application/json">)(.*?)(</script>)', re.DOTALL)
# Search for the script tag that contains the JSON array
match = script_pattern.search(content)
if not match:
await update.message.reply_text(f"Title '{title}' not found in the file.")
return None # Return None if no matching script tag is found
# Extract the JSON string from within the <script> tag
json_str = match.group(2)
# Load the JSON content
try:
tiddlers = json.loads(json_str)
except json.JSONDecodeError as e:
print(f"Error decoding JSON: {e}")
return None
# Find the tiddler with the matching title
modified = False
for tiddler in tiddlers:
if tiddler.get("title") == title:
# Append the new text to the "text" field
tiddler["text"] += f" {new_text}"
modified = True
break
if not modified:
await update.message.reply_text(f'Title "{title}" not found in the file.')
return None # Return None if no matching title is found
# Convert the modified tiddlers back to a JSON string
updated_json_str = json.dumps(tiddlers, indent=2)
updated_json_str = updated_json_str.replace("<", "\\u003C")
# Reinsert the modified JSON back into the <script> tag
modified_content = content.replace(json_str, updated_json_str)
overwrite_drive_file(modified_content)
await update.message.reply_text(f"Title '{title}' has been updated successfully.")
def overwrite_drive_file(modified_text):
# Write the modified text to a temporary file
service = get_drive_service()
temp_file_path = 'temp_modified_file.html'
with open(temp_file_path, 'w') as temp_file:
temp_file.write(modified_text)
# Create a MediaFileUpload object for the new content
media = MediaFileUpload(temp_file_path, mimetype='text/html')
# Use the Google Drive API to update the file content
updated_file = service.files().update(
fileId=FILE_ID,
media_body=media
).execute()
# Optionally, clean up the temporary file after upload
os.remove(temp_file_path)
return updated_file
async def edit_a_edit(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
title = "a_edit"
new_text_with_a = update.message.text
new_text = new_text_with_a[2:]
await modify_text_in_html(update, title, new_text)
async def edit_b_edit(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
title = "b_edit"
new_text_with_b = update.message.text
new_text = new_text_with_b[2:]
await modify_text_in_html(update, title, new_text)
def main():
# Set up logging
logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', level=logging.INFO)
# Create the application and pass your bot's token.
application = Application.builder().token(TELEGRAM_TOKEN).build()
# Add handler for the /getfile command
application.add_handler(MessageHandler(filters.Regex('(?i)^a .*'), edit_a_edit))
application.add_handler(MessageHandler(filters.Regex('(?i)^b .*'), edit_b_edit))
application.add_handler(CommandHandler("getfile", get_file))
# Start the bot
application.run_polling()
if __name__ == '__main__':
main()
This is all I personally need for this project, but I’d certainly be interested for the sake of fun and curiosity in anything more general you were thinking of, @paotsaq (or anyone else), that would work across saving methods now that I have two very specific programs for two very specific saving methods.
2 Likes
Can something be done for wiki saved in onedrive?