Using the query string to carry extra instructions

Developers
, ,
#1

I’m thinking of trying to use the query string inside my wiki. I’m wondering if there is any prior art, or if there are strong arguments against doing this. It would be used to set the initial value of a tiddler used to control which content is shown. That value could be changed with some control (radio buttons, I’m thinking.)

The wiki http://scott.sauyet.com/Tiddlywiki/WIP/AndoverCharterChanges/ is only partially complete, but it should be enough to show the idea. It is going to track some analysis of proposed changes to my town’s legal charter. The first stepis to get the text of the charter—now buried in an image-based PDF and hard to read—into a readable wiki with granular tidders. So far I’ve converted two of the ten sections, and I think I still need a little work on my outlining technique, but it’s not too bad.

There are many proposed revisions to the current charter embedded like this:

image
image1233×578 145 KB

The yellow highlight is for existing language proposed for removal. The red text is proposed to be added. (It’s not my choice of color schemes – don’t blame me! :grinning_face_with_smiling_eyes: )

I would like to be able to toggle between three states: (1) Show the existing charter text, (2) Show the proposed text, or (3) Show them both… as in this screenshot. I should be able to handle that easily enough. But I also want to be able to share links to the wiki in one of these states. The permaview syntax is powerful, but I don’t think it will help me do this. That’s why I’m thinking of highjacking the url query.

It’s there, it’s easily queryable from JS. I think I’ll be able to set it too, on radio button change .

I believe I might have urls like

    https:my-wiki.example.com/?current
   <!--                       ^^^^^^^^          -->

or

    https://ny-wiki.example.com/?propsed#Charter%2F2022%2FII%2F203%2FB
    <!--                        ^^^^^^^^                           -->

or with the query parameter of mixed. I can put that value on body or perhaps the story river.

I don’t know the details of how to do the rest of this, but it all sounds simple enough. However, I haven’t seen people use the query string this way. And maybe there’s a good reason.

For me, this will be served as a single-file wiki, so there’s no back-end to worry about.

So, do you think the fundamental design is broken? Should I skip it? If not, any advice for how to achieve a similar effect without making and maintaining mutiple copes of the wiki.

1 Like
TOC macro for a forest, not a tree
New Site: Page to explain charter changes
#2

According to the URI syntax specification it should be OK

But

We should also have TW spec in the view. As soon as you do something and it is published you basically did define a “de-facto” standard and that may cause compatibility problems if the core decides to do something similar.

If you send eg: https://tiddlywiki.com/?test=asdf#HelloThere to tw-com it is perfectly fine. TW at the moment will ignore the query string and it will show the HelloThere tiddler.

So it should be possible, that you define a state in the query string, that prepares the tiddler you want to show.

It only would need a marker that identifies the query-string as the one for you. eg:

https://tiddlywiki.com/?vnd=ss-v1&state=current#HelloThere

With this system you could define your logic, that it analyses the first key=value pair as a vendor string. eg: vnd=something

If your logic can identify this vendor string, it will handle it.
If it does not know it, it will ignore it and hand over the whole thing to the next query-parser, which may know it.

So the logic itself would need to be a plugin that uses the TW-hook mechanism. Because th-xxx hooks can stack.

vnd=ss-v1 stands for vnd … vendor, ss … Scott Sauyet -v1 first version of your handler. the -v1 will make it easy for you to fail silently.

If your own handler can not identify the version string it should ignore the whole thing and hand it over to the next handler …

With the vendor identifier we can make sure, that we have a mechanism, that several plugin authors can define different query strings without compatibility problems.

eg: I could use https://tiddlywiki.com/?vnd=wl-v1&something=else&one=more#HelloThere

just some thoughts
Mario

1 Like
#3

Hi Scott, this is a need that comes up fairly often, that is to be able to pass data to a wiki on startup via the URL. There is a preference to do so via the location hash as opposed to the query as the latter is usually reserved for server usage.

This use case is one of the underlying reasons for the desire to move the processing of the hash and related startup behaviour into wikitext.

Some of my plugins currently use the presence of ##readonly=true in the location hash to trigger read only mode. This takes advantage of the core ignoring anything after a ## in the startup handling, the official share plugin is another example of this.

Here are some notes that I have from a previous preliminary brainstorming session:

  • pass everything after a double hash ## to startup actions, potentially using the double hash or ampersand character as a separator.
  • support key value pairs and encode them as JSON before passing to startup actions
  • Examples: ##key=value##key1=value1##key2=value2##key3=a,b,c,d,e,f or ##key=value&key1=value1&key2=value2&key3=a,b,c,d,e,f
  • low level wikitext mechanism to take over startup handling, including navigation
  • a tagged mechanism specifying handlers in wikitext where each one tries to handle the incoming parameters, and unless stop propagation is specified the subsequent handlers process the parameters too. Ordering of the handlers will matter.
#4

Yes, thanks. That wasn’t something I was worried about. I’ve used query parameters from JS before, and looked this stuff up often enough. The following is what I was worried about:

I was very tired when writing that post and was afraid I didn’t get across the concern. But this captures it admirably.

I’m not particularly concerned about setting a de facto standard, but I am worried about forward compatibility, in this case more for philosophical reasons than practical ones. Your suggestion looks like a good way to get around that.

It seems to me that it would be better to allow mixing and matching of different sources’ query parameters. Perhaps it would be better to prefix each one?

https://tiddlywiki.com/?vnd-ss-v1_state=current#HelloThere
https://tiddlywiki.com/?vnd-tw-v3_prop1=abc&vnd-tw-v3_prop2=xyz&vnd-ss_v1_state=current#HelloThere

And if we do this, the prefix vnd- is unnecessary, so this would be a little simpler:

https://tiddlywiki.com/?ss-v1_state=current#HelloThere
https://tiddlywiki.com/?tw-v3_prop1=abc&tw-v3_prop2=xyz&ss-v1_state=current#HelloThere

One other thought would be that TiddlyWiki would reserve the prefixes tw-<version>_ for itself (and possibly all two-letter combos starting with t, the way it does with tc for CSS class names.) And I would propose that my-<version>_ be reserved for use directly in wikis, so that plugins or the core would never interfere with userland.

Does that make sense?

Yes; I’m sure you know that I’ve been actively involved in one of the motivating bits.

That was the main reason I raised this issue. In my case, the server is just a plain HTTP server; it knows nothing about Tiddlywiki. But clearly it could run on Node (which is how I’m developing it) or in many other configurations. The question is whether I would break forward compatibility in doing this.

I am worried about doing it through the location hash, though. Perhaps your expanded handling will be able to deal with it, but I am concerned about this information surviving changes to the permalink/permaview. By updating the query string, I bypass that concern.

I think @pmario’s ideas of a vendor prefix would mesh nicely with this, possibly with my alternative above.

What I didn’t mention in my post is that this is very short-term. I want to solve it in the next day or two. The wiki and that show-hide behavior are only important before an election on the 5th of November. The goal of this wiki is to provide analysis and recommendations regarding the proposed charter changes; those will be decided in the election. I will probably keep the outline version (once I’ve cleaned it up) for the long-term, but it won’t have the before-after annotations.

So in one very important sense, forward compatibility is not crucial at all. But I do prefer to consider it.

What I could not possibly have mentioned in that post—I think I fell asleep three times trying to complete it!—was how I see this idea falling in TiddlyWiki in general. I think it ends up between what can be done with BrowserStorage plugin and what can be done by updating a wiki or making a copy. It allows me to share with others, via URL, a view of the wiki differing from the default view, but only in ways provided by that wiki. So instead of telling someone, “Visit https://mywiki.example.com/, then choose the “proposed” link in the right side-bar”, I can just say “Visit https://mywiki.example.com/?my-v1_version=proposed”.

With BrowserStorage, I could set it up so that my user can choose that default for themselves. That makes sense for color schemes, for some layout options, and many other features. But it doesn’t make sense here, where shared urls would very much be meant to show a particular version of the main content.

I could also use a build step to create three different versions of the page, which varied only in the content of one tiddler but showed these three views. That might be my fallback position, if I can’t get that done. But it’s much less flexible, both for me and for users.

Finally, this got two very different answer from two of the luminaries of talk. @EricShulman, care to break the tie? Or should we bring this to Daddy and ask @jeremyruston to chime in? :wink:

#5

I think in the short term you can really go with either a query string or using the ## in the location hash and it should not matter too much. In terms of future proofing, future TiddlyWiki server side implementations such as MWS may one day use the query string for their own purposes and serving the same wiki on such a sever may prove problematic. However, based on your use case, that seems to be a highly unlikely scenario.

In terms of leveraging prior art, the share plugin is meant precisely to share a wiki with some additional payload via the URL. The additional payload in turn can be config tiddlers that influence what the viewer sees. So if you don’t mind the convoluted URLs that this will entail, you might be able to use that out of the box. Here is an example that hides the sidebar by default, by passing the relevant state tiddler in the URI.

@Scott_Sauyet do you also need the user to be able to switch between the multiple views, and for the URL to update correspondingly?

#6

In TiddlyWiki Classic, you could specify “paramifiers” on the URL, using # as a prefix (see https://classic.tiddlywiki.com/#StartupParameters). And, as of TW2.4.2, this was extended so that if a paramifier began with #chk or #txt, it would be automatically added to the internal config.options.* array of settings.

-e

#7

Ok. I will investigate how the share wiki does its work. It’ll be good to know even if I don’t use it.

Yes. But I think I can just do that in a JS macro/proc/widget/(which?) with something like:

history.replaceState(null, null, '?my_v1-version=' + version + document.location.hash)

and if I care to be more robust, by using URLSearchParams to keep any other query parameters.

(Note that I switched underscores and hyphens. I think I prefer that to the above.)

Wow, I’d forgotten about that! I vaguely remember copying someone’s code using #chk to achieve some (now cringe-worthy) animation on startup. Better left forgotten though, as it wouldn’t work with TW5! Thanks for the memory.

#8

(Confessional note: I’m replying partly for selfish reasons: I want to be part of this conversation thread because techniques like this are central to what I’m doing with educational LMS-integrations. I suspect nothing below is new or surprising to you!)

I am using a system that constructs automated hash-enhanced urls, so that each student is loading a customized iframe (based on the LMS’s access to various strings specific to each student). But as you note, there’s brittleness in any solution that requires the location hash to stay the same!

I solve this problem by having a startup script that parses the hash components, generating some temp tiddler values based on those hash strings… Then those temp-tiddler values do the actual work, from that point forward, for the duration of the session (until reload). Is that technique of potential use in your case?

1 Like
#9

tiddler values == titles?

#10

generating one temporary tiddler, with its field values determined by the hash-string details.

(To be clear, it’s a startup action that generates the same tiddler title every time, but sets field values in accord with the incoming hashes (which are different for each student, once they’re in the LMS iframe). That specific temp tiddler is then polled by various other system tiddlers, templates, etc., so that certain things “automagically” appear, customized to each student.)

#11

I might do something similar with the query values. But it’s only a partial solution. I want user-changed values to be stored in the URL even when they change permalinks/views. While I’m separately pursuing work to change the way permalinks/views are handled, that’s turning out to be a laborious process. This one needs to be agile.

#12

I got this working, at least for a first pass.

You can see this at

http://scott.sauyet.com/Tiddlywiki/WIP/AndoverCharterChanges/v3/

If you play with the toggle in the Overview, you can get one of these URLs, instead:

http://scott.sauyet.com/Tiddlywiki/WIP/AndoverCharterChanges/v3/?atc-version=before

or

http://scott.sauyet.com/Tiddlywiki/WIP/AndoverCharterChanges/v3/?atc-version=after

(It’s hard to see the differences between Before and After without a close reading of certain parts of the text. But there’s a big visual difference between the default version (Compare) and either of the others.)

Note that there is no server side interpreting that query parameter; you get served identical content. It’s all happening in the browser.

The process is roughly this:

If you press one of the radio buttons in Version Toggle (found also in the sidebar, and transcluded into Overview), we toggle several classes on the body tag. We have either hide-before, hide-after, or neither—which happens when you choose Compare. This is used by the CSS to hide and show various parts of the document. There’s nothing particularly noteworthy about the CSS, but if you want to see it, it’s in $:/_/atc/styles/css.

This also adds a key/value to the document query string, here atc-version to “before”, “after” or blank, where I think of atc-, or possibly atc_v1- / atc_v2-, as my own prefix (Andover Town Charter), so that plugins and other tools could also use the query string without stepping on one another’s toes.

On startup, a StartupAction looks for that query string key. If it’s there, we set the field used by those radio buttons to the value. And we branch on the value. Each branch toggles those CSS classes on the body.

That’s the system. The relevant parts are:

which are used to add and remove classes from the body element,

which add and remove a key/value pair from the URL query string,

and

which are (1) an action to handle the startup, searching the query string for my token, then based on the result, adding and removing classes from the body, and setting the field used by the radio widgets; and (2) a $:/tags/StartupAction/Browswer tiddler that does nothing to but call that last widget.

The only other part of interest here is Version Toggle, but that is a straightforward mini-form, with calls on radio-button selection to the actions above.

Questions:

  1. Are the add/remove body class tiddlers a solved problem? Am I reinventing the wheel? Is this already available in wikitext?

  2. The actions all use $-prefixed variable names, as I was using a core widget as a model. Are such variable names meant to be used only by the core?

  3. I wanted to perform the other necessary actions inside the startup action, ideally globalizing (or importing) the actions inside Toggle Version and calling them from here. But I couldn’t get it to work, which is why I have action-addcharterversion. It feels wrong, though. Did I just not struggle enough? Should that work? Or does a startup action only call action widgets directly inside itself, and not those imported? (Even if have a solution to this, I may not use it. This is meant to be a one-off, and now that I have my infrastructure, I really need to focus on the whole point of this, which is an analysis of the ballot questions themselves.)

More minor question:

  1. (Probably only for JS developers): When removing all the query parameters, I can’t actually remove the leading ? There’s a commented-out line showing one of my attempts; with it, the ? disappears, but the last remaining query parameter remains in place. If you uncomment that one and comment its neighbor, after saving and reloading you can see the problem using the buttons in a test class I forgot to remove, Body class list. This is probably not a TW question, but if you know something about history.replaceState and URLSearchParams, you may have a suggestion. It may also be impossible to remove. I’m curious but not terribly worried.

  2. (Mostly for CSS folks, and extremely minor) When I change the body class by toggling the radio buttons, the tiddlers which become empty should still show, but I would like for their background color to revert to their default (white in my vanilla theme), which I try with background-color: inherit. But instead, they’re the same as the story-river background, #f4f4f4. I can easily use background-color: white or figure out the correct palette entry, but I would like to know why. inherit works fine for the other properties I revert. As I said, it’s minor, but it is perplexing.

1 Like
#13

Just a short note that safe mode can be invoked in the url already

Safe mode is enabled in the browser by starting TiddlyWiki with the URL hash set to the string #:safe. For example:

https://tiddlywiki.com/#:safe

I have long thought introducing some hackability so designers can use this mechanism and define additional keywords that trigger some actions would be an elegant extension of this existing method.

  • This would be a form of keyword driven startup actions.
  • Having actions triggered when a keyword is not provided would also be useful, eg without a particular keyword use read only view, and to reverse something set with a previouse use of a keyword.

This kind of hackability may prove useful on top of MWS @jeremyruston

1 Like
#14

Thank you. I knew that this existed. I didn’t know how it was triggered.

While this is slightly different from my own implementation, it’s clearly in the same spirit.

#15

I think query string is a good way to pre-install some tiddler, for example, set the $:/layout to some alternative layout, like Calendar layout, and assign a date range to a state tiddler, so you can easily share a calendar to others.

Maybe this can works like “Share” core plugin?

#16

The design may not be broken, but the technique of using query strings in this way does have a significant pitfall that would prevent its use by the core.

The problem is that according to the spec, although query strings are visible in the browser they are owned by the server. So, for example, if a TiddlyWiki document is stored on MS SharePoint using the WebDAV saver, the SharePoint server will see the query strings as being parameters to the page request (alongside any query strings that it uses itself eg for authentication). It may throw an error or ignore them, but it is unlikely to pass them along so that they are still visible to code running in the page (ie TiddlyWiki).

So, using query strings in this way is generally safe only if you control the server, and know that the code is never going to be run on another server that doesn’t support query strings.

#17

I did suspect that this was a no-go for the core, but to me the issue would be more that there’s no realistic scheme to avoid using keys some server doesn’t already claim for itself or that some server might choke and throw an error on unexpected keys or key/value combinations.

I don’t think this is a concern:

It may throw an error or ignore them, but it is unlikely to pass them along so that they are still visible to code running in the page (ie TiddlyWiki).

At least when we’re talking about browsers, the server does not have a choice. The browser is told to visit a URL; the query is part of that URL. And, while that is passed to the server, the server can’t alter the URL set in the browser, except to offer a redirect. While it could do that or could throw an error, it seems unlikely, and in a widely-used server tool like Sharepoint, that is presumably at least configurable.

Maybe the work around the hash value will fix this. @saqimtiaz’ suggestion of ##readonly=true could well be the fix, but, for my use-case at least, that would have to survive changes to permalinks/views.

For the moment, I’m fine. This wiki should be done(-ish) soon and it’s only useful through the November 5 election. But I hope that we can address the problem of specifying certain loading behavior through the URL, something orthogonal to the current use of the hash for delineating the initial tiddler(s).