@Scott_Sauyet just some information about json that assisted me,
First and foremost the json Mangler plugin. I have a utility wiki set up I use to import data sets from different sources and create a data plugin which I then use where needed.
If you drag and drop a tag pill between wikis you will notice it allows you too import the tiddlers so tagged. To achive this, each tiddler is in effect converted to a json tiddler inside the import.
I have created a button, that appears on any tiddler containing a particular filter, this button can be dragged and dropped on any wiki and import all the tiddlers named in the filter. This is fundimentaly creating a package of json representations of tiddlers.
In many ways all off the above are doing the same as the titles of this topic.
Perhaps, but I don’t think any of them will apply to my scenario. The user enters a name into a text box on a tiddler, copies some code to the clipboard, enters that code on the console of an entirely different (non-TW) site, and then copies the result from that external console.
The user now has a string of JSON, and I would like to import it back into my current wiki.
That’s what I was asking about. After they paste this string into another text box, how can I convert that to a new tiddler and open it in edit mode? I had a working solution, but it meant manually copying every field. Saq helped me write a much more elegant version, and Mario helped me remove the remaining redundancy.
I don’t think any of the techniques you suggested will help with that. But they are clearly useful in many cases. Thank you for sharing them!
If you stop to think about the solutions above, to work, they need to generate the JSON representation of tiddlers, you are trying to create, and then allows them to transfer into any wiki.
No, I don’t understand your motivation to take the approach you are. But my leads above and here, can lead you to your solution.
If you describe the desired functionality, without part of the method, I may be able to give you a more direct solution.
If that string of json is in “tiddler format”, and contains all the field/values pairs, it is trivial to import it into any wiki. To achieve what I did above, I use the template and thus macro designed to convert tiddlers to JSON.
Yes, you could build an approach where you paste into a text field of a tiddler a JSON representation you can import, or generate a plugin, but this is not as easy as other approaches.
If that string defines a tiddler (containing further JSON tiddler definitions) you can also put meta data in the outer tiddler, and do all kinds of tricky things, generate a plugin, place import conditions and more.
To drag and drop, or export tiddlers, we use $:/core/templates/exporters/JsonFile see https://tiddlywiki.com/#jsontiddler%20Macro which is javascript $:/core/modules/macros/jsontiddlers.js “Macro to output tiddlers matching a filter to JSON”
This uses a function TiddlersAsJson defined in $:/core/modules/wiki.js
This function iterates the fields to create a JSON definition of a tiddler.
I thought I did a reasonable job of that in the OP, but here’s another try:
I want to create my Member tiddlers mostly out of data available in our intranet site. There is an API to capture the desired information, using the member’s name. But to access that API, you need to be logged into the company single-sign on. Even if it were possible to do this with a stand-alone TW, the amount of paperwork required and the ridiculous amount of time it takes are prohibitive. Also, the other administrators of this new wiki are TW novices. I want to give them a simple-as-possible solution to gather this information and use it to create a new tiddler.
How would you suggest handling that?
It is in tiddler format, and it contains all the important field/value pairs. There are some still left blank, as the sources of information are elsewhere than this intranet wiki. But it is a string containing a JSON object.
I’d love to know how. Right now that process involves pasting the JSON into a text box and hitting a button. The code that supports this is what Saq helped me clean up. But even so, I don’t find it trivial, not at all. Do you have a better process? (The only other alternative I could come up with was to have the user save this JSON to a file, drag that file to the wiki, choose the “Import” button, and then select the newly generated tiddler to enter edit mode. That seems much more of a hassle than what I have so far.)
and paste it into the box in Step 4. (Steps 1 - 3 have to do with how you would really fetch this data from our company intranet; they are somewhat interesting to me, but not really relevant here, unless you have a technique that somehow would let me skip them.)
Now Fred should show up in the Members list.
Then hit the button in Step 5. If you want an avatar, you can drag this image in:
and rename it to FredFlintstone.jpg and Fred will have a picture wherever pictures show up.
I understand this JSON object is crafted to be a JSON array, not the Tiddlers definition? See my example below
Lets say we have a place you can drop the JSON that was generated (with a custom importer), or an edit field, as you have already, into which you could paste it;
Here is an example to play with; I call the “tiddlers JSON structure”
I understand you already, Paste the something similar to the above into a tiddlers text field,
give it the type “application/json”
give it plugin-type of plugin
Save it and now you have access to the tiddler within “this plugin”
Of course the above can be automated further like paste here and press button to create the plugin tiddler, or even better a special drop zone to do the same.
Here is one approach like your current one.
Currently if I copy the sample text above and ctrl-V on a wiki it imports it as an untitled tiddler, in plain text. Perhaps the import process could detect it begins with;
{
"tiddlers": {
Then offer a button to import it/them with, the type “application/json”, plugin-type of plugin.
This would be the users workflow;
Go to the site where you construct this input code and copy it to the clipboard.
Go to the target wiki and ctrl-v, In the import tiddler that appears, a button will appear saying “import as tiddlers”, they press this and they are done?
Others may be;
Rather than have the user copy the content from the site, have the content published at an address and use http put/get, external tiddler or other methods to obtain this.
Generate a bookmarklet, of the source data, that can then be clicked to apply it to the destination wiki. You can create tiddlers this way, without further interaction, or use the data protocol.
If you craft it as a tiddler containing a link to the function, once imported you can click to import.
To make a tiddlywiki drag package, such as when you drag a tag to another wiki, this works because tiddlywiki crafts the content in a way it can be dropped, imported in a target wiki, if you design your source wiki to do this (although you don’t have handy the Javascript in tiddlywiki) you could make a tool to craft the content suitably and make it draggable. Tiddly wiki won’t know it did not come from a wiki.
It may be as simple as using the “tiddlers JSON structure”, making the “content draggable” on the source site.
I know the above does not include a detailed solution, to have made choices then provided a full solution would take a lot of time and may be not what you wanted anyway.
No, it’s a simple, singular JSON object, not an array – not even an array containing only one entry – just an object. I do understand the plugin JSON format too, but I have no reason for that. I don’t want my Members in a plugin, and after my initial batch load, we will always be loading one at a time.
Note that this format is perfectly understood by TW. If you pasted the JSON snippet from my previous answer into a file and dragged that file to a wiki, it will offer to import a tiddler exactly as described.
That’s an interesting alternative, but I don’t think it’s any more simple than the current one. The user will already have started from my Add Member tiddler in order to get the information needed on the other site, so pasting in there and hitting a button seems at least as easy as pasting somewhere on the page and hitting a custom import button.
(I may still play with this, as I haven’t done any sort of custom import work, and it might be a good learning experience, but I don’t expect it to be any simpler than what I have so far.)
That site is not in my control, and I’ve already mentioned that it is behind an authentication wall. It just happens to have a useful API that my colleagues can use from the developers’ console.
There is no static source data. It’s specific to the new member I want to add. So I can’t craft a bookmarklet for that data, not in a manner that isn’t already much more complex that the current technique. But do you know if a bookmarklet can access the clipboard? I’ve never tried. If it can, then that could be an interesting alternative to the paste-here-then-click-there technique, as we could skip the paste step altogether. Of course if I can do that, presumably I can do it in the wiki as well.
The source system is not in my control, is not TiddlyWiki, is not even any sort of wiki, Essentially, it’s the company directory entry for an employee/contractor, backed up by a JSON web service I am currently using to extract data and convert to a JSON-stringified object representing a tiddler.
No, I wouldn’t expect a detailed solution. Hints are great. But I don’t think any of these are headed in a useful direction, not for the problem I’ve asked about. Still, they may be useful to me another day.
Only to put stuff in the clipboard, not pull from the clipboard. However the bookmarklet can act as a clipboard if it contains data, and this can be installed as a tiddler in a wiki, as if it were pasted (silently).
Since bookmarklets are in fact JavaScript functions that run in the browser, what ever you can do with a JavaScript function in the current window, you can enhance to call tiddlywiki functions, and create tiddlers containing data in or obtained by that function. The bookmarlet needs to be “hit” or triggered of course.
If you login to your third party site and access something in the page, more than likely a javascript function can address, or even harvest it. I bit like the concept of a screen scraper.
I tend to use ChatGPT to determine if such things are possible and even to write them. For example the following confirms what I suggest is possible;
Can a javascript function in a browser window capture content into a temporary location we can use later?
Yes, a JavaScript function running in a browser window can capture content and store it in a temporary location for later use. This can be achieved through various mechanisms, depending on the scope and lifetime you want for the stored data. Here are the most common methods:
Variables in Memory: You can store data in variables within your JavaScript code. This data will be available as long as the page is not reloaded or the browser tab remains open. However, this is the most temporary form of storage, as closing the tab or reloading the page will clear the data.
LocalStorage: The localStorage web API provides a way to store data across browser sessions. Data stored in localStorage persists even when the browser is closed and reopened. It’s suitable for storing small amounts of data that needs to be accessed across multiple sessions.
SessionStorage: Similar to localStorage, the sessionStorage API allows you to store data for the duration of the page session. The data is cleared when the session ends (i.e., when the tab is closed). This is a bit more persistent than variables but not as long-lasting as localStorage.
IndexedDB: For more complex or larger amounts of data, IndexedDB is a low-level API for client-side storage of significant amounts of structured data. This can include files/blobs. IndexedDB is more powerful and allows for more complex data storage than localStorage or sessionStorage, including the ability to store data in a more structured format.
Cookies: Cookies can also be used to store data, but their primary purpose is to store data that needs to be sent back to the server on subsequent requests. Cookie data is included with every HTTP request, making it suitable for tracking or session data but less efficient for large quantities of data that only need to be client-side.
Cache API: Part of the Service Workers API, the Cache API allows you to programmatically cache responses to requests. While primarily intended for offline use cases, it can be used to temporarily store network request results for later use.
In-Memory Databases: JavaScript libraries that implement in-memory databases (e.g., LokiJS) can also be used to temporarily hold and query large amounts of data more efficiently than the built-in storage options.
Each of these methods has its own set of advantages and limitations in terms of capacity, persistence, and scope (i.e., whether the data is available only to a particular tab/window or shared across the entire browser). The choice of which to use depends on the specific needs of your application, such as the amount of data, how long it needs to be stored, and whether it needs to be available across sessions or tabs.
Notice across windows/tabs is not excluded, but no details are given.
I suspect we could even have a Javascript dialogue prompt for you to paste content into it, then save this as a tiddler on a tiddlywiki. I have done something similar already. That is you use the bookarklet to replace the import/paste process.
The work flow would be obtain your 3rd party data, and go to your wiki and hit the bookmarklet (or function/button in your wiki) it pops up a dialogue you paste into, hit OK, and it creates the data as a tiddler or tiddlers.
I may experiment with this, with the help of Chat GPT.
Hi, why not drag and drop the json to the wiki? We share many json tiddler exported from tiddler’s export view template button in the Chinese forum and QQ group.
Or use “import” from “tools” sidebar tab, can also import json file too.
The user doesn’t have a file to drag and drop, just a string on the clipboard. Of course we could have the user create a file, but I’d prefer it to be simpler than that. And I think it is now, with the solution I created and Saq and Mario helped improve.
But that was the point: Can we find a way to do this without a file? The answer is yes, and it’s not so ugly as my first version.
That’s what I figured. I’ve written and used bookmarklets extensively over the years, and I never saw any clipboard reading. I apologize: that’s a question I should have looked up and not posted here. I know that you’re a strong advocate of the use of bookmarks with TW. I haven’t done anything with that; most of my wikis are written to be used by other people. By myself as well, to be sure, but not only by myself. Asking users to install bookmarks to use with my wiki seems… unlikely.
I did consider screen scraping, but when I realized that the page was running on top of an API, and that I could just call that API from the console, I was able to move past that idea. Now it’s just a matter of calling that API and converting the result into my tiddler format, a relatively simple task in JS. (If you’re curious, the code that does this is copied from $:/_/my/data/js-code/inside-info with something like fetchInfo('Fred Flintstone') appended. The tiddler JSON from my earlier message will be returned in the console. That’s what the user can cut and paste back into my tiddler.
While yes, I still would love to learn other options for this overall workflow, the question is mostly about how – once they’ve gotten the tiddler JSON onto their clipboard, how can I turn it into an actual tiddler?
I think that’s missing an important clause: “…we can use later from another domain?”
All these techniques are restricted to working within your current domain, for very sound security reasons: You don’t want your online gaming site reading information from your banking site.
Most (all?) of them will work across tabs/windows, but not across domains.
That would be almost comparable in simplicity to my current solution. But that “almost” is important. It offers no advantage and requires the user to first install the bookmark.
That’s a great alternative. I’ve used a little drag-and-drop on TW, but not much, and have not seen this, even though I knew something like this was possible.
I have to decide now. My approach is a little more friendly to the TW novice. But this one would better help ease novices into the TW way. A nice dilemma to have.
I didn’t read thru everything, but a dropzone seems to fit your purpose well, especially if the content is already in TW JSON format. If not, you can write your own custom JavaScript deserializer module to further process the dropped string into tiddler/s.
There is a hack, probably not the simple way you are looking for, to create tiddlers using json string by creating your own “$:/Import” tiddler with a specific tiddlers json string format and sending a “tm-perform-import” message to complete the import.
I did something similar also using an input box to capture a json string (nicely formatted by a bookmarklet) and import as tiddlers. I shared it here (Copy info from web site to tiddlywiki using bookmarklet and json tiddler dropzone), look under “JSON TIDDLER INPUT BOX”. The wikicode shared there doesn’t seems like an improvement over yours.
Thank you. It looks like john.edw_gmail.com beat you to this by a few hours. I will investigate tonight whether I’d rather try this or keep to the original solution, with Saq’s and pmario’s fixes. But this is certainly worth considering. I also like the idea of moving the deserialization into a single place instead of running part of it on the external site and the rest in the code discussed above.
Coming to this late. But I’m running two variants of this kind of solution, and want to add to the discussion in case it’s helpful:
One variant is kookma’s refnotes solution, which takes pasted bibtex json data, using a custom dropzone (or “paste zone” in practice for me), which you can find here: Refnotes 1.9.0 — references, citations, footnotes and abbreviations in Tiddlywiki … and transforms in various ways to translate into the TiddlyWiki field-structure that refnotes uses (prefixing most fields with bibtex- etc.).
The other takes jsons exported by my university’s (= your alma mater’s ) moodle LMS, and makes various transformations on these (since the json I can export from moodle doesn’t even come with a title field, for example), so they can be imported. See discussion thread with key post here:
It absolutely is helpful. These seem similar to two other recent suggestions. The three of them, taken together, seem to make me want to reconsider my approach, and try a drop-zone with all the conversion happening inside TW instead of in the console of that intranet site. I’ll have to think about it more, though, as I believe the dropzone approaches might require more user steps than my current one. That’s not all bad, though, as some of those steps might help them get more familiar with TW. So, more to ponder!
Thank you very much for your insight and your examples!
I am not trying to persuade you to any approach, but for completeness, and future readers.
I wonder if tiddlyclip could help?
Remember this would replace what ever triggers the API and no console access needed. It could be done once.
We do not have to install bookmarks to use them, you can have links (in a website or tiddlywiki) that executes the javascript function, or move it into a custom action widget.
You may be able to include the API call in such a function.
Bookmarks are very easy to install if needed. Drag an’ drop.
Can you call this from tiddlywiki directly? We have other solutions that do.
Actually that is what I implied by “use later”, and in fact the clipboard already acts like that.
Keep in mind if you trigger the function in your browser, its different from if the site attempts to, this is where bookmarklets operate above the contents of a particular domain. They can operate across domain/tab/window, and tend to act on the current one, but as the user we choose which is the current one.
Perhaps If we used a browser extension, “Service Workers” (one example) we could make a storage location in which we can store something and both push and pull to/from it, to assist with cross site automation. Extensions like this can be given cross “domain access”.
This was a feature in Timimi for a while, now depreciated.
Not necessarily, the function can be triggered inside the destination wiki, even a button. No need to see the import or edit a text field and paste.
This seems to me it could be a very useful function - combining paste and import in a single step. I may make this myself.
Yes, build the logic and control where you have the most influence, and do as little as possible, where you do not, eg the other site. this way you can always improve the process.
I don’t know; I’ve heard a lot about it, but never used it, or even investigated its capabilities.
I’ve been using bookmarklets since they were called favelets. I do understand how they work. I have dozens on both personal and work browsers. I’ve never written to the clipboard from them, but can see how that would be useful. But the console is also useful, and a lot more flexible. I will consider this, but am not happy with the idea of telling users, “Drag this link to your bookmarks bar (after unhiding it if it’s hidden). Then, to import a member go to the Inside intranet page, click that bookmarklet, enter the member’s name at the prompt, then when it’s visible, hit another button to copy the result to the clipboard.”
I’m not sure that last is necessary. I expect it would be, given Firefox’s security model around the clipboard, but my users will likely be using Chrome. The point is that entering the name and clicking the button would trigger an AJAX request, and the data would be available only in the asynchronous callback, which likely would not be what FF considers inside a “short running user-generated event handler”, so another user action would be necessary. But this is from memories a decade old; this may have changed.
It would have to run from the intranet page to be useful, so yes, the user would have to install them in my case.
I mentioned in the OP and in at least two previous responses to you that I cannot, not without a hugely painful integration with company single-sign-on authentication. Even if TW could support that, it’s quite unlikely that the company would grant permission for it.
Well, the examples were Variables in Memory, LocalStorage, SessionStorage, IndexedDB, Cookies, the Cache API, and In-Memory Databases. I’m pretty sure none of those works cross-domain. If they did, it would be quite frightening.
I’m wary of bookmarklets for my users. There’s no way I would look to a browser extension!
I could see this being very useful!
Well, I’m talking about where the logic is run, not where it’s maintained. Right now the logic to call the service and transform the result into TW format is maintained in the tiddler $:/_/my/data/js-code/inside-info, but this is combined with the name we’re adding and copied to the clipboard, to run in the console of the intranet page. I could instead use a <$dropzone> as others have been suggesting, and a custom deserializer that does that work on input.
So I’m building and maintaining the logic in the only place I have any influence. But I am running it elsewhere. I have been looking at these techniques, and, while not completely decided, am leaning to keeping the original approach.
Its good you have an acceptable answer and i will leave it there. Only you know about your source server. I do think some of your assumptions or understanding of my suggestions are not altogether accurate, nor do I expect you to run with my hunches, only some of which I have tested.
you also face the need to work with your system users I dont know their capacities.
) moodle LMS, and makes various transformations on these (since the json I can export from moodle doesn’t even come with a title field, for example), so they can be imported. See discussion thread with key post here:
this way you can always improve the process.