How to use python script simulate http request to open a certain tiddler?

Ori · November 8, 2021, 2:10pm

I want to use fzf to search for the tiddler and send the search result back to tw by python.

I check this post https://groups.google.com/g/tiddlywiki/c/7RF7L-A0v3g. But in javascript.

I know that I can check the network in the developer tools to get the http request information.

Here is what I get when I want to open a tiddler named ScratchCameraEffect

Here is the code I make to

import requests
headers = {
    'Accept': '*/*',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8,en-US;q=0.7,ja;q=0.6,zh-TW;q=0.5',
    'Connection': 'keep-alive',
    'Content-Length': '216',
    'Content-type': 'application/json',
    'Host': '127.0.0.1:8080',
    'Origin': 'http://127.0.0.1:8080',
    'Referer': 'http://127.0.0.1:8080/',
    'sec-ch-ua': '"Google Chrome";v="95", "Chromium";v="95", ";Not A Brand";v="99"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Windows"',
    'Sec-Fetch-Dest': 'empty',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Site': 'same-origin',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36',
    'X-Requested-With': 'TiddlyWiki',
}
payload = {
    "created": "20211108134301421",
    "title": "$:/StoryList",
    "text": "",
    "fields": {
        "list": "ScratchCameraEffect"
    },
    "modified": "20211108134301421",
    "type": "text/vnd.tiddlywiki"
}

url = 'http://127.0.0.1:8080/recipes/default/tiddlers/%24%3A%2FStoryList'
ret = requests.put(url, data=payload, headers=headers)

When I run the python script:
I get the error requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, remote shutdown the current connection
And also the tw crushed.

So I ask for help why this happened? Seems my py script make the request same as what I get in the developer tool.

Rafael · November 9, 2021, 3:52am

I know that it’s not exactly what you’ve asked… But wouldn’t the permaview syntax be enough for your case?

https://tiddlywiki.com/#:[[PermaLinks]][[HelloThere]][[Introduction to filter notation]][[Simple ways to write protect tiddlers]][[AlertMechanism]]

This solution would be opening a new tab with the Tiddlers that you’ve filtered… You could open it from the command line with something like:

firefox 'https://tiddlywiki.com/#:[[PermaLinks]][[HelloThere]][[Introduction to filter notation]][[Simple ways to write protect tiddlers]][[AlertMechanism]]'

Or in Python:

import os
os.system("firefox 'https://tiddlywiki.com/#:[[PermaLinks]][[HelloThere]][[Introduction to filter notation]][[Simple ways to write protect tiddlers]][[AlertMechanism]]'")

Ori · November 9, 2021, 6:49am

Thank you. It worked.
But not exactly what I want, I need change the opened tw and quicker.
Still thank you.

Ori · November 9, 2021, 7:06am

Now I make it done. I forgot to stringify the json data, which is mentioned in the post above. My mistake.

The correct code is here, use json.dumps to stringify the json data.

import requests
import json

headers = {
    'Accept': '*/*',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8,en-US;q=0.7,ja;q=0.6,zh-TW;q=0.5',
    'Connection': 'keep-alive',
    'Content-Length': '216',
    'Content-type': 'application/json',
    'Host': '127.0.0.1:8080',
    'Origin': 'http://127.0.0.1:8080',
    'Referer': 'http://127.0.0.1:8080/',
    'sec-ch-ua': '"Google Chrome";v="95", "Chromium";v="95", ";Not A Brand";v="99"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Windows"',
    'Sec-Fetch-Dest': 'empty',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Site': 'same-origin',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
                  '(KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36',
    'X-Requested-With': 'TiddlyWiki',
}

payload = {
    "created": "20211109063935937",
    "title": "$:/StoryList",
    "text": "",
    "fields": {
        "list": "StyleTagIcon"
    },
    "modified": "20211109063953241",
    "type": "text/vnd.tiddlywiki"
}

url = 'http://127.0.0.1:8080/recipes/default/tiddlers/%24%3A%2FStoryList'
ret = requests.put(url, data=json.dumps(payload), headers=headers)
# ret = requests.put(url, data=payload, headers=headers)

But seems things still not like what I expected.

After the python script executed, tiddler what I want not open immediately. You have to reload the page manually. Which take 3-4 for me to wait for the tw page loaded again.
After refresh the tw page, what I opened just now is all closed just leave the one I opened by python script which is still unaccepted.

Any way to make it?

I am already to modify the source code tiddlywiki by myself . LOL. Start from nearly zero knowledge of js.

Ori · November 9, 2021, 2:20pm

I found that I can change the storylist.tid from outside to restore the opened tiddlers.

Content of storyList:

created: 20211109141804513
list: PythonScriptToOpenTiddlerFromCommandlineWorkflow IconSearch WSLPretty
modified: 20211109141804513
title: $:/StoryList
type: text/vnd.tiddlywiki

You can shut down the node (I use nodejs version of tw btw),
then modify the storylist.tid, second line change the value of “list”
Then reload the page,
But still it needs reload, which take some time I can’t accept…

jeremyruston · November 9, 2021, 4:32pm

Hi @Ori you’ve touched on two limitations of TW5 at present:

Modifying tiddler files directly requires the Node.js server to be restarted before the changes will be processed
When using the HTTP API to make changes to the store, takes up to 60 seconds for changes to propagate to all connected browsers

One workaround for the 2nd problem is to set a shorter polling interval – see https://tiddlywiki.com/#Hidden%20Setting%3A%20Sync%20Polling%20Interval

joshuafontany · November 10, 2021, 6:52am

You can also use the Server Cloud drop down & click Sync from Server (which will call the appropriate method immediately).

Ori · November 10, 2021, 9:49am

Can you give me a example? I don’t know what is drop down & click Sync?

Here is my code, the fields ---> list ---> value can be multiple tiddler names that will make tw open all of them. And current opened tiddlers you can read from the storylist.tid. So just one question left:
How to make the Response to the request immediately .

import requests
import json

headers = {
    'Accept': '*/*',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8,en-US;q=0.7,ja;q=0.6,zh-TW;q=0.5',
    'Connection': 'keep-alive',
    'Content-Length': '216',
    'Content-type': 'application/json',
    'Host': '127.0.0.1:8080',
    'Origin': 'http://127.0.0.1:8080',
    'Referer': 'http://127.0.0.1:8080/',
    'sec-ch-ua': '"Google Chrome";v="95", "Chromium";v="95", ";Not A Brand";v="99"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Windows"',
    'Sec-Fetch-Dest': 'empty',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Site': 'same-origin',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
                  '(KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36',
    'X-Requested-With': 'TiddlyWiki',
}

payload = {
    "created": "20211109072538620",
    "title": "$:/StoryList",
    "text": "",
    "fields": {
        "list": "DvorakQWERTY IconSearch"
    },
    "modified": "20211109072538620",
    "type": "text/vnd.tiddlywiki"
}

url = 'http://127.0.0.1:8080/recipes/default/tiddlers/%24%3A%2FStoryList'
ret = requests.put(url, data=json.dumps(payload), headers=headers)

saqimtiaz · November 10, 2021, 11:59am

Perhaps the Server Sent Events work that got split off into a plugin would be helpful here, it immediately propagates changes from the server to the cleint.

@pmario do you know what the current status of that is and whether it is currently usable?

Ori · November 10, 2021, 12:12pm

The setting of polling interval not work for me.

And before set the polling interval my tw will not reload automatic no matter how long I wait for it. I have to reload manually. After I set it to 10s, not work either same with before.

The situation is :
I make the request, then waiting … nothing happens, after I reload the page manually. The Request work.

And one more question is the polling same with reload?
I don’t want reload the whole page, cuz it take 3~4 sec, maybe I have too many tiddlers now.

pmario · November 10, 2021, 3:48pm

I didn’t read the thread at all. … nothing of it.

SSE is a plugin from Arlen22 atm. see: GitHub - twcloud/tiddlyweb-sse … It’s a bit bumpy.

If the server gets a tiddler from a client with the default PUT mechanism it sends an SSE to all subscribed clients. … The clients then do the same thing as if they where triggered by the timeout.

That way it was possible to keep it backwards compatible, but from my point of view it’s a hack and has much room for improvement. …

BUT it works as intended. So you pretty much can see “live editing” if 2 tabs opened the same wiki and a nodejs server.

The important thing is, that the change has to be sent with a PUT request to the server.