Creating a tiddler from a JSON *string*

#1

Background

I’m creating a wiki to manage the chess club I help run at GigantiCorp, where I work. It’s going well, and I’m quite happy with the results. You can see a (somewhat anonymized1) version of it at http://scott.sauyet.com/Tiddlywiki/Demo/ClubDemo/v1/. I am proud to have been able to create this without asking a single question about it here. I’m clearly gaining deeper TW knowledge. But now it’s time for clean-up and optimization, and I’m ready to ask for help. This is the first of a few questions I expect to ask.

But I also would love other feedback on the wiki: Are there UI features that seem poorly done or ill-conceived? Are there implementation details that could be significantly simplified? I haven’t really tried for “pretty” yet, but can you suggest changes to improve the look?

1 I’ve cleaned up names, headshots, anything sensitive from GigantiCorp (such as ids, emails, etc,), and private email addresses. But the site connects users to Lichess, and I have left intact the user’s lichess ids, which, I suppose, could reveal other user details. Those ids are necessary for various functioning of the site.

Question

Is there a simple way to create a tiddler from a string of JSON (not a file) generated by another system and pasted in by the user.

I know I could ask the user to drop this to a file, something.json, and drag that to the wiki, but I’m hoping to avoid that. I know that in JS there’s the $tw.Tiddler constructor and $tw.wiki.addTiddler, and I certainly can use that if nothing else is available. But I was wondering if this is already exposed through the wiki’s own tools.

I have a version, but it’s a little ridiculous, copying every property from the input JSON to a field in the new tiddler:

<$button>
 <$action-sendmessage $message="tm-new-tiddler" 
    title={{{ [{$:/temp/create-member!!inside-profile}jsonget[title]] }}} 
    tags={{{ [{$:/temp/create-member{!!inside-profile}jsonget[tags]] }}}  
    avatar={{{ [{$:/temp/create-member!!inside-profile}jsonget[avatar]] }}}
    coached-by={{{ [{$:/temp/create-member{!!inside-profile}jsonget[coached-by]] }}} 
    comments={{{ [{$:/temp/create-member!!inside-profile}jsonget[comments]] }}} 
    ... plus 15 more fields ...
/> 
  Create member
</$button>

You can see this in http://scott.sauyet.com/Tiddlywiki/Demo/ClubDemo/v1/#Add%20Member.

(Yes, I know this could be cleaned up by caching a reference to [{$:/temp/create-member{!!inside-profile}] but I’m hoping to eliminate this boilerplate altogether.)

Can I avoid those ridiculous shenanigans and somehow simply create a tiddler based on this JSON string?

Context

When I create a user, I want to fetch a number of tiddler fields from the corporate intranet directory. That directory is available as JSON, and I’ve already used it to batch-load the 90+ current members, converting that structure to my desired tiddler format, and then dragging a file with the result to my wiki.

I collected this data by running a script in the developer tools console on that intranet directory page, where I’m already logged in. This avoids having to deal with any authentication issues. To add a new member, I personally could just simply repeat this process, with a simpler script only designed to fetch one person’s data.

But I would rather that the other club administrators don’t have to do the whole file shuffle. I have a screen they can use, where they enter the person’s name, and get a script to run in the console on that intranet directory page, copy the resulting JSON, and paste that back in a second entry field and click a Create button, which uses that JSON to build a tiddler. It’s a bit clunky, but workable. But I manually parse every field out of the input, and that seems ridiculous.

Obviously TW handles this every time we drag a JSON file onto the page. But I don’t know if that capability is exposed without descending into JS.

Other options

I don’t really like the process I’ve developed. I’m open to other ideas. The constraints are simple:

  • The other administrators may have less technical backgrounds; I want the process to be reasonably simple for them
  • I do not want – ever! – to try to integrate with company auth and single-sign-on. So, as far as I can tell, I can’t fetch the data through the new-ish HTTP capability in TW.
  • I like having this as a stand-alone wiki, for all the usual reasons, so I wouldn’t want to try fetching this data dynamically (even if I could get through the auth issues); I really want a one-time import process to create the user.

Still, the current process does work, and the club administrator who adds most users to the current spreadsheet was able to use it. So unless someone has a brilliant alternative idea, I’m willing to stick with this. But I would feel so much better without all the JSON => field manipulation.

Plugin idea: Read/write Obsidian's new open "JSON Canvas" format
#2

See Can TiddlyWiki read external tiddlers from URL parameter? - #27 by saqimtiaz and in particular the usage of action-set multiple fields. Precede that with action-delete to avoid merging incoming fields into an existing tiddler.

1 Like
#3

Thank you. That works perfectly. I had seen setmultiplefields before, and it always looked to be more hassle than it was worth, trying to align fieldnames and values in two different lists. But I had only thought about it for static lists. For auto-generated dynamic ones it’s exactly what was needed.

The following is how I’ve used it:

<$button>
  <$action-setmultiplefields 
    $tiddler="[{$:/temp/create-member!!inside-profile}jsonget[title]]" 
    $fields="[{$:/temp/create-member!!inside-profile}jsonindexes[]]" 
    $values="[{$:/temp/create-member!!inside-profile}jsonindexes[]] :map[{$:/temp/create-member!!inside-profile}jsonget<currentTiddler>]"
  />
  <$action-navigate $to={{{ [{$:/temp/create-member!!inside-profile}jsonget[title]] }}} />
  <$action-sendmessage $message="tm-edit-tiddler" $param={{{ [{$:/temp/create-member!!inside-profile}jsonget[title]] }}}/>
  Create member
</$button>

Two follow-up questions, if you have the time:

  • Is there a better way to combine the setting of the fields and the opening of the new tiddler in the story river in edit mode? My prior tm-new-tiddler solution did this in one step. This one uses setmultiplefields, navigate to, and tm-edit-tiddler: three steps. Is there a way to use this multiple-fields technique with one or two steps?

  • Is there a simple rule about when to use "[some[filter] +[goes[here]]" versus {{{ [some[filter]] +[goes[here]] }}} in parameters? In the above snippet, I was able to use the simpler quotes in the the first action, but needed the triple curly braces for the latter two, else I get that filter string as the title. I can’t understand why that is, especially as the contents are the same ($tiddler in the first, $to in the second, $param in the third.)

These questions are far from crucial. This solution works, and is much cleaner than what I started with, but inquiring minds want to know.

1 Like
#4

I am on the road so this will be brief, but you should be able to use the genesis widget to create an action send message widget so you can still use tm-edit-tiddler. Genesis also allows passing two lists, one for keys and the other for values.

#5

Thank you very much for taking time for this during your travels!

Ok, I’ve so far steered away from the genesis widget. Perhaps it’s time to investigate it.

Thank you very much for your help!

– Scott

#6

You don’t need “navigate to” – tm-edit-tiddler is enough. So 2 steps will work.

#7

Well, I swear I tried that… several times. But it works just fine this time! Sometimes I just don’t know about that brain of mine!

Thank you very much. This feels much nicer.

#8

@Scott_Sauyet just some information about json that assisted me,

First and foremost the json Mangler plugin. I have a utility wiki set up I use to import data sets from different sources and create a data plugin which I then use where needed.

If you drag and drop a tag pill between wikis you will notice it allows you too import the tiddlers so tagged. To achive this, each tiddler is in effect converted to a json tiddler inside the import.

I have created a button, that appears on any tiddler containing a particular filter, this button can be dragged and dropped on any wiki and import all the tiddlers named in the filter. This is fundimentaly creating a package of json representations of tiddlers.

In many ways all off the above are doing the same as the titles of this topic.

Ask if you want more details

#9

Perhaps, but I don’t think any of them will apply to my scenario. The user enters a name into a text box on a tiddler, copies some code to the clipboard, enters that code on the console of an entirely different (non-TW) site, and then copies the result from that external console.

The user now has a string of JSON, and I would like to import it back into my current wiki.

That’s what I was asking about. After they paste this string into another text box, how can I convert that to a new tiddler and open it in edit mode? I had a working solution, but it meant manually copying every field. Saq helped me write a much more elegant version, and Mario helped me remove the remaining redundancy.

I don’t think any of the techniques you suggested will help with that. But they are clearly useful in many cases. Thank you for sharing them!

#10

If you stop to think about the solutions above, to work, they need to generate the JSON representation of tiddlers, you are trying to create, and then allows them to transfer into any wiki.

No, I don’t understand your motivation to take the approach you are. But my leads above and here, can lead you to your solution.

  • If you describe the desired functionality, without part of the method, I may be able to give you a more direct solution.

If that string of json is in “tiddler format”, and contains all the field/values pairs, it is trivial to import it into any wiki. To achieve what I did above, I use the template and thus macro designed to convert tiddlers to JSON.

  • Yes, you could build an approach where you paste into a text field of a tiddler a JSON representation you can import, or generate a plugin, but this is not as easy as other approaches.
  • If that string defines a tiddler (containing further JSON tiddler definitions) you can also put meta data in the outer tiddler, and do all kinds of tricky things, generate a plugin, place import conditions and more.

To drag and drop, or export tiddlers, we use $:/core/templates/exporters/JsonFile see https://tiddlywiki.com/#jsontiddler%20Macro which is javascript $:/core/modules/macros/jsontiddlers.js “Macro to output tiddlers matching a filter to JSON”

  • This uses a function TiddlersAsJson defined in $:/core/modules/wiki.js
  • This function iterates the fields to create a JSON definition of a tiddler.
#11

I thought I did a reasonable job of that in the OP, but here’s another try:

I want to create my Member tiddlers mostly out of data available in our intranet site. There is an API to capture the desired information, using the member’s name. But to access that API, you need to be logged into the company single-sign on. Even if it were possible to do this with a stand-alone TW, the amount of paperwork required and the ridiculous amount of time it takes are prohibitive. Also, the other administrators of this new wiki are TW novices. I want to give them a simple-as-possible solution to gather this information and use it to create a new tiddler.

How would you suggest handling that?

It is in tiddler format, and it contains all the important field/value pairs. There are some still left blank, as the sources of information are elsewhere than this intranet wiki. But it is a string containing a JSON object.

I’d love to know how. Right now that process involves pasting the JSON into a text box and hitting a button. The code that supports this is what Saq helped me clean up. But even so, I don’t find it trivial, not at all. Do you have a better process? (The only other alternative I could come up with was to have the user save this JSON to a file, drag that file to the wiki, choose the “Import” button, and then select the newly generated tiddler to enter edit mode. That seems much more of a hassle than what I have so far.)

You can see this process by visiting

http://scott.sauyet.com/Tiddlywiki/Demo/ClubDemo/v2/#Add%20Member

then copy this text:

{
    "title": "Fred Flintstone",
    "tags": "Member",
    "avatar": "FredFlintstone.jpg",
    "coached-by": "",
    "comments": "",
    "first-name": "Fred",
    "full-name": "Fred J Flintstone",
    "last-name": "Flintstone",
    "lichess-id": "",
    "personal-email": "",
    "trav-dept": "Cranes",
    "trav-div": "Bedrock",
    "trav-email": "fflintstone@example.com",
    "trav-id": "n12345",
    "trav-lob": "Construction",
    "trav-location": "Bedrock, ST, USA",
    "trav-profile-url": "https://example.com/n12345",
    "trav-short-name": "fflintstone"
}

and paste it into the box in Step 4. (Steps 1 - 3 have to do with how you would really fetch this data from our company intranet; they are somewhat interesting to me, but not really relevant here, unless you have a technique that somehow would let me skip them.)

Now Fred should show up in the Members list.

Then hit the button in Step 5. If you want an avatar, you can drag this image in:

FredFlintstone

and rename it to FredFlintstone.jpg and Fred will have a picture wherever pictures show up.

#12

I understand this JSON object is crafted to be a JSON array, not the Tiddlers definition? See my example below

Lets say we have a place you can drop the JSON that was generated (with a custom importer), or an edit field, as you have already, into which you could paste it;

Here is an example to play with; I call the “tiddlers JSON structure”

{
    "tiddlers": {
        "sticky editor toolbar css": {
            "created": "20240131053536228",
            "text": ".tc-tiddler-preview {\n  overflow: unset;\n}\n.tc-editor-toolbar {\n\tposition: -webkit-sticky;\n\tposition: -moz-sticky;\n\tposition: -o-sticky;\n\tposition: -ms-sticky;\n\tposition: sticky;\n\ttop: 40px;\n\tbackground: #f3f3f3;\n\tz-index: 500;\n}",
            "tags": "$:/tags/Stylesheet",
            "title": "sticky editor toolbar css",
            "modified": "20240131053647234"
        }
    }
}

I understand you already, Paste the something similar to the above into a tiddlers text field,

  • give it the type “application/json”
  • give it plugin-type of plugin
  • Save it and now you have access to the tiddler within “this plugin”

Of course the above can be automated further like paste here and press button to create the plugin tiddler, or even better a special drop zone to do the same.

Here is one approach like your current one.

Currently if I copy the sample text above and ctrl-V on a wiki it imports it as an untitled tiddler, in plain text. Perhaps the import process could detect it begins with;

{
    "tiddlers": {

Then offer a button to import it/them with, the type “application/json”, plugin-type of plugin.

This would be the users workflow;

  • Go to the site where you construct this input code and copy it to the clipboard.
  • Go to the target wiki and ctrl-v, In the import tiddler that appears, a button will appear saying “import as tiddlers”, they press this and they are done?

Others may be;

  • Rather than have the user copy the content from the site, have the content published at an address and use http put/get, external tiddler or other methods to obtain this.
  • Generate a bookmarklet, of the source data, that can then be clicked to apply it to the destination wiki. You can create tiddlers this way, without further interaction, or use the data protocol.
    • If you craft it as a tiddler containing a link to the function, once imported you can click to import.
  • To make a tiddlywiki drag package, such as when you drag a tag to another wiki, this works because tiddlywiki crafts the content in a way it can be dropped, imported in a target wiki, if you design your source wiki to do this (although you don’t have handy the Javascript in tiddlywiki) you could make a tool to craft the content suitably and make it draggable. Tiddly wiki won’t know it did not come from a wiki.
    • It may be as simple as using the “tiddlers JSON structure”, making the “content draggable” on the source site.

I know the above does not include a detailed solution, to have made choices then provided a full solution would take a lot of time and may be not what you wanted anyway.

#13

No, it’s a simple, singular JSON object, not an array – not even an array containing only one entry – just an object. I do understand the plugin JSON format too, but I have no reason for that. I don’t want my Members in a plugin, and after my initial batch load, we will always be loading one at a time.

Note that this format is perfectly understood by TW. If you pasted the JSON snippet from my previous answer into a file and dragged that file to a wiki, it will offer to import a tiddler exactly as described.

That’s an interesting alternative, but I don’t think it’s any more simple than the current one. The user will already have started from my Add Member tiddler in order to get the information needed on the other site, so pasting in there and hitting a button seems at least as easy as pasting somewhere on the page and hitting a custom import button.

(I may still play with this, as I haven’t done any sort of custom import work, and it might be a good learning experience, but I don’t expect it to be any simpler than what I have so far.)

That site is not in my control, and I’ve already mentioned that it is behind an authentication wall. It just happens to have a useful API that my colleagues can use from the developers’ console.

There is no static source data. It’s specific to the new member I want to add. So I can’t craft a bookmarklet for that data, not in a manner that isn’t already much more complex that the current technique. But do you know if a bookmarklet can access the clipboard? I’ve never tried. If it can, then that could be an interesting alternative to the paste-here-then-click-there technique, as we could skip the paste step altogether. Of course if I can do that, presumably I can do it in the wiki as well.

The source system is not in my control, is not TiddlyWiki, is not even any sort of wiki, Essentially, it’s the company directory entry for an employee/contractor, backed up by a JSON web service I am currently using to extract data and convert to a JSON-stringified object representing a tiddler.

No, I wouldn’t expect a detailed solution. Hints are great. But I don’t think any of these are headed in a useful direction, not for the problem I’ve asked about. Still, they may be useful to me another day.

Thank you!

#14

Only to put stuff in the clipboard, not pull from the clipboard. However the bookmarklet can act as a clipboard if it contains data, and this can be installed as a tiddler in a wiki, as if it were pasted (silently).

  • Since bookmarklets are in fact JavaScript functions that run in the browser, what ever you can do with a JavaScript function in the current window, you can enhance to call tiddlywiki functions, and create tiddlers containing data in or obtained by that function. The bookmarlet needs to be “hit” or triggered of course.
  • If you login to your third party site and access something in the page, more than likely a javascript function can address, or even harvest it. I bit like the concept of a screen scraper.

I tend to use ChatGPT to determine if such things are possible and even to write them. For example the following confirms what I suggest is possible;

Can a javascript function in a browser window capture content into a temporary location we can use later?

Yes, a JavaScript function running in a browser window can capture content and store it in a temporary location for later use. This can be achieved through various mechanisms, depending on the scope and lifetime you want for the stored data. Here are the most common methods:

  1. Variables in Memory: You can store data in variables within your JavaScript code. This data will be available as long as the page is not reloaded or the browser tab remains open. However, this is the most temporary form of storage, as closing the tab or reloading the page will clear the data.

  2. LocalStorage: The localStorage web API provides a way to store data across browser sessions. Data stored in localStorage persists even when the browser is closed and reopened. It’s suitable for storing small amounts of data that needs to be accessed across multiple sessions.

  3. SessionStorage: Similar to localStorage, the sessionStorage API allows you to store data for the duration of the page session. The data is cleared when the session ends (i.e., when the tab is closed). This is a bit more persistent than variables but not as long-lasting as localStorage.

  4. IndexedDB: For more complex or larger amounts of data, IndexedDB is a low-level API for client-side storage of significant amounts of structured data. This can include files/blobs. IndexedDB is more powerful and allows for more complex data storage than localStorage or sessionStorage, including the ability to store data in a more structured format.

  5. Cookies: Cookies can also be used to store data, but their primary purpose is to store data that needs to be sent back to the server on subsequent requests. Cookie data is included with every HTTP request, making it suitable for tracking or session data but less efficient for large quantities of data that only need to be client-side.

  6. Cache API: Part of the Service Workers API, the Cache API allows you to programmatically cache responses to requests. While primarily intended for offline use cases, it can be used to temporarily store network request results for later use.

  7. In-Memory Databases: JavaScript libraries that implement in-memory databases (e.g., LokiJS) can also be used to temporarily hold and query large amounts of data more efficiently than the built-in storage options.

Each of these methods has its own set of advantages and limitations in terms of capacity, persistence, and scope (i.e., whether the data is available only to a particular tab/window or shared across the entire browser). The choice of which to use depends on the specific needs of your application, such as the amount of data, how long it needs to be stored, and whether it needs to be available across sessions or tabs.

  • Notice across windows/tabs is not excluded, but no details are given.

I suspect we could even have a Javascript dialogue prompt for you to paste content into it, then save this as a tiddler on a tiddlywiki. I have done something similar already. That is you use the bookarklet to replace the import/paste process.

  • The work flow would be obtain your 3rd party data, and go to your wiki and hit the bookmarklet (or function/button in your wiki) it pops up a dialogue you paste into, hit OK, and it creates the data as a tiddler or tiddlers.
  • I may experiment with this, with the help of Chat GPT.
#15

Hi, why not drag and drop the json to the wiki? We share many json tiddler exported from tiddler’s export view template button in the Chinese forum and QQ group.

Or use “import” from “tools” sidebar tab, can also import json file too.

#16

The user doesn’t have a file to drag and drop, just a string on the clipboard. Of course we could have the user create a file, but I’d prefer it to be simpler than that. And I think it is now, with the solution I created and Saq and Mario helped improve.

But that was the point: Can we find a way to do this without a file? The answer is yes, and it’s not so ugly as my first version.

#17

This worked with a cut and paste of the Fred Flintstone data

<$dropzone deserializer="application/json" tag="div">
<div style.height="200px" style.width="200px" style.border="2px solid red"  style.textAlign="center" style.lineHeight="200px" tabindex="1">
Paste or Drop corporate intranet directory data here.
</div>
</$dropzone>

Paste Json data from Clipboard.tid (368 Bytes)

#18

That’s what I figured. I’ve written and used bookmarklets extensively over the years, and I never saw any clipboard reading. I apologize: that’s a question I should have looked up and not posted here. I know that you’re a strong advocate of the use of bookmarks with TW. I haven’t done anything with that; most of my wikis are written to be used by other people. By myself as well, to be sure, but not only by myself. Asking users to install bookmarks to use with my wiki seems… unlikely.

I did consider screen scraping, but when I realized that the page was running on top of an API, and that I could just call that API from the console, I was able to move past that idea. Now it’s just a matter of calling that API and converting the result into my tiddler format, a relatively simple task in JS. (If you’re curious, the code that does this is copied from $:/_/my/data/js-code/inside-info with something like fetchInfo('Fred Flintstone') appended. The tiddler JSON from my earlier message will be returned in the console. That’s what the user can cut and paste back into my tiddler.

While yes, I still would love to learn other options for this overall workflow, the question is mostly about how – once they’ve gotten the tiddler JSON onto their clipboard, how can I turn it into an actual tiddler?

I think that’s missing an important clause: “…we can use later from another domain?”

All these techniques are restricted to working within your current domain, for very sound security reasons: You don’t want your online gaming site reading information from your banking site.

Most (all?) of them will work across tabs/windows, but not across domains.

That would be almost comparable in simplicity to my current solution. But that “almost” is important. It offers no advantage and requires the user to first install the bookmark.

#19

That’s a great alternative. I’ve used a little drag-and-drop on TW, but not much, and have not seen this, even though I knew something like this was possible.

I have to decide now. My approach is a little more friendly to the TW novice. But this one would better help ease novices into the TW way. A nice dilemma to have.

Thank you very much!

#20

I didn’t read thru everything, but a dropzone seems to fit your purpose well, especially if the content is already in TW JSON format. If not, you can write your own custom JavaScript deserializer module to further process the dropped string into tiddler/s.

There is a hack, probably not the simple way you are looking for, to create tiddlers using json string by creating your own “$:/Import” tiddler with a specific tiddlers json string format and sending a “tm-perform-import” message to complete the import.

I did something similar also using an input box to capture a json string (nicely formatted by a bookmarklet) and import as tiddlers. I shared it here (Copy info from web site to tiddlywiki using bookmarklet and json tiddler dropzone), look under “JSON TIDDLER INPUT BOX”. The wikicode shared there doesn’t seems like an improvement over yours.

The json string have to be in the “$:/Import Tiddler Temporary Format” ( How do TWs JSON Formats Look Like ) and look like this:

{
    "tiddlers": {
        "New Tiddler": {
            "title": "New Tiddler",
                --- other fields stripped for readability --- 
        },
        "Hello World": {
            "title": "Hello World",
                --- other stripped for readability --- 
        }
    }
}