Bulk import local images into single tiddler as links

[clsturgeon]

@arunnbabu81

Thanks for your reply and for trying to help. I would be happy to test those batch files.

Here is a summary of the usage of the batch file (order does not matter):

twImageReport -w YourTWfile.html -f path\to\images

where 

-w is the wiki file. If missing it will locate the newest HTML file in the folder. 
-f is the folder of images. If missing it will look at all folders. All subfolders are reviewed as well.

It produces three files.

1. a text file with a list of image files that were found in a folder with a reference to it in the TW HTML file.

2. a text file with a list of image files that were found in a folder with no reference to it in the TW file.

3. a json file that can be used to import tiddlers that populates the _canonical_uri field with a relative path to the image files. The json contains new tiddler definitions for those missing image file references.

Note: It will not find the references with forward slashes (/) in the path (which are commonly used), rather it only matches where the path has backslashes (); a quirk that might be worth fixing.

Here is a demo where I used the -f switch, but not the -w switch.:

Download the bat file here (in a zip file)
https://clsturgeon.github.io/MemoryKeeper/support/downloads/twImageReport.zip

Be sure to backup–everything!

Craig

[Mohammad]

Hi @jacng
I like your method and I call it private method. It allows us to have a TW on thumb drive and just drag and drop the required bookbarkelrt ot bookmark tab of browser and then it start to work.
No permission, no admin (specially for work), no extenstion, …
So, I think your method worths to be improved and bundled as a bookmarker/import tool and hopefuly later as a clipper!

[jacng]

Thanks! Actually it wasn’t intentional nor my idea I needed sample bookmarklets to illustrate the json tiddler dropzone and borrowed the idea on how to present bookmarklets n Tiddlywiki from this thread Is there a mime type or deserialiser for JavaScript function like in bookmarklets - #21 by btheado

[arunnbabu81]

@clsturgeon thanks for sharing this. I will try it out when I am on my window PC

[Mark_S]

This is a dropzone version of the bulk image importer that uses the output of jacng’s bookmarklet after capturing a file directory. You hover and click on the dropzone, then paste (ctl-v). This way you get the advantage of the bookmark but the flexibility of wikitext, but without needing an entry box.

Hopefully it’s not too brittle. It turns out the import process json-stringifies things, but there’s no reverse filter operator. So I had to have the code do it’s own clean-up of the import tiddler contents.

There’s 3 variables you can set. One to set the pre-amble path you want to apply to each entry. The next the name of the tiddler that will receive the links. “Depth” refers to how many levels of the original path you want to retain. You have to have at least depth “1” – that’s the file name.

Currently it just filters for .png and .jpg.

   path="file:///home/mark/data/Wikis/test2/"
   basetitle = "My Image Links"
   depth="2"

Hopefully this adds value to the project. But if not … oh well

Incoming.json (1.3 KB)

[Mohammad]

Hi Craig,
Very useful automation!
Minor comments:

1. Is it possible to add other image type most importantly svg?
2. Is it possible to optionally turn off the echo? I mean ifor sake of speed .bat file work silently and optionally user can turn on display on screen?

[Mohammad]

@clsturgeon

Download the bat file here (in a zip file)
https://clsturgeon.github.io/MemoryKeeper/support/downloads/twImageReport.zip

I tried the code with folder contains space in title like C:\TW\03. Resources\00. Images\Noto-emoji-2020-09-16-unicode13_1\png\128

The twImageReport. create path in log files and json file like mages\Noto-emoji-2020-09-16-unicode13_1\png\128 I assume it cannot handle spaces!

[jeremyruston]

@Mark_S

the import process json-stringifies things, but there’s no reverse filter operator

I’m not sure if a jsonunstringify operator would be generally useful but in the meantime you can unstringify JSON strings by wrapping them in double quotes and using the jsonget operator. For example:

[[Hello \" there]addsuffix["]addprefix["]jsonget[]]

[clsturgeon]

@Mohammad

I assume it cannot handle spaces!

Try with double quotes at start and end of the parameter.

[clsturgeon]

@Mohammad

• Is it possible to add other image type most importantly svg?
• Is it possible to optionally turn off the echo? I mean ifor sake of speed .bat file work silently and optionally user can turn on display on screen?

Yes, both are easy to add. I’ll have a look when I get to my pc.

[Mohammad]

@clsturgeon

Try with double quotes at start and end of the parameter.

Actually my issue report was on the basis of using path with spaces in double quotes, I called the te

twImageReport -w ext_images.html -f "C:\TW\03. Resources\00. Images\Noto-emoji-2020-09-16-unicode13_1\png\128"

What I see in JSON and log files is like mages\Noto-emoji-2020-09-16-unicode13_1\png\128

[jacng]

Hopefully it’s not too brittle. It turns out the import process json-stringifies things, but there’s no reverse filter operator. So I had to have the code do it’s own clean-up of the import tiddler contents.

Looking at the codes, that seems to be referring to { } " " characters being quoted wrongly? How to duplicate the error ? i use the following json tiddler text generated by the bookmarklet, the original text 'TEST "QUOTATION" TEST { CURLY BRACKET }' is imported corrected.

[{"text":"TEST \"QUOTATION\" TEST { CURLY BRACKET }","type":"text/vnd.tiddlywiki","title":"(@bookmark) DropZone Testing — a non-linear personal web notebook","tags":"import@bookmark","fdir.name":"file:///C:/Downloads/TW_525_empty%20Dropzone%20testing-old%20FileDir%20Viewer%20(pixel).html","fdir.fullpath":"file:///C:/Downloads/TW_525_empty%20Dropzone%20testing-old%20FileDir%20Viewer%20(pixel).html","fdir.filedate":"20230410141457000"}]

The bookmarklet simply uses javascript 's built-in function to JSON.stringify the contents. Is there anything that can be done to correct at the bookmarklet’s end also ?

Oh, in the “Incoming” tiddler, there is an extra “x” before the deserializer. The dropzone works if “x” is removed, otherwise it’s importing as plain text.

<$dropzone xdeserializer="application/json" importTitle="dropme" class="mydropzone" actions=<<dropzone-actions>>  >

[Mark_S]

The problem isn’t the bookmarklet. It’s the import process. If you import your text created by the bookmarklet, you get this in the $:/import (dropme) tiddler

{
    "tiddlers": {
        "Untitled text/plain": {
            "title": "Untitled text/plain",
            "text": "{\"tiddlers\":{\"(file) /home/mark/data/Wikis/test2/files/6vegetables-that-are-the-same-plant\":{\"text\":\"\",\"type\":\"text/vnd.tiddlywiki\",\"title\":\"(file)

The part I want is after “text”

"{\"tiddlers\":{\"(file) /home/mark/data/Wikis/test2/files/6vegetables-that-are-the-same-plant\":{\"text\":\"\",\"type\":\"text/vnd.tiddlywiki\",\"title\":\"(file) ...

You can see that all the quote marks are proceeded by a back slash. I presume this is because the following text has been json-stringified. So I have to roll this back. But there’s no equivalent de-json-stringify operator, though there must be one internally in the core.

If you do this by hand, and press the “import” button, the values imported are ok. But if you have to do that, there’s no point to using the dropzone for automation.

Edit: In this context, creating the “dropme” tiddler, I tried it with and without the deserializer and it made no difference. Presumably because the text string is still treated as a string and not as it’s own JSON object.

[jeremyruston]

@Mark_S

But there’s no equivalent de-json-stringify operator, though there must be one internally in the core

See my answer above; de-json-stringifying can be done by evaluating the JSON string with jsonget

[clsturgeon]

@Mohammad

Actually my issue report was on the basis of using path with spaces in double quotes

EDITED: The script was not designed where the -f parameter is an absolute path. It was designed to be a relative path, relative to the location of html file. Having said that, I tested using your folder structure. However, I could not determine which folder your html file resides in for you. I placed it in the TW folder. Regardless, the absolute path seemed to work for me, as the script works out the relative path to place in the json file.

Can you provide more information? What is the error?

Make sure the batch file is in the same folder as the HTML file, and when you run the script, your working directory (current directory at the command prompt) is the same as the HTML and the batch file.

Redownload the same zip file. I made 1 change; added support for SVG. I have not yet added a quiet/silent switch. I was able to replicate any issues with spaces or embedded periods in the folder names or the image file names.

https://clsturgeon.github.io/MemoryKeeper/support/downloads/twImageReport.zip

Thank you for reporting the issue.

Craig

[clsturgeon]

@clsturgeon

Make sure the batch file is in the same folder as the HTML file, and when you run the script, your working directory (current directory at the command prompt) is the same as the HTML and the batch file.

Here is a quick tip (you may or may not know). In Windows Explorer, navigate to the folder you want. In the address bar type in “cmd” and press Enter. The command prompt will appear with the working directory already set for you.

[arunnbabu81]

I think I will finally settle for this solution by buggyj using tiddlyclip- Bulk import local images · buggyj/tiddlyclip-plugin · Discussion #120 · GitHub

Using this I can even select the which all images in the folder needs to be imported. I can create rules for importing images in [[]] and [img[]] formats. Also I dont have to specify from which folder I have to import in the rules. I just have to open the file directory in firefox, select the images and import them.

[Mohammad]

@clsturgeon

The script was not designed where the -f parameter is an absolute path. It was designed to be a relative path, relative to the location of html file.

Noting the point, the code works as expected! Thank you

[Mohammad]

Hi @clsturgeon
Thank you it works now. If I put the images folder in the same folder of html and bat file, then any level of nesting with space in folders name work!
In my previous example, html and bat file were on D: drive, but the image where on C: drive!

I just made a small change to the code

set "tags=$:/tags/ExternalImage"
		set "endPath=!relPath:\=\\!"
		if "!first!"=="true" (
			echo {^"text^":^"^",^"title^":^"!filename!!extension!^",^"type^":^"image/!filetype!^",^"_canonical_uri^":^"!endPath!^",^"tags^":^"!tags!^"} >>"%~dp0twImagejson.json"
			set "first=false"
		) else (
			echo ,{^"text^":^"^",^"title^":^"!filename!!extension!^",^"type^":^"image/!filetype!^",^"_canonical_uri^":^"!endPath!^",^"tags^":^"!tags!^"} >>"%~dp0twImagejson.json"
		)

I added the tags=$:/tags/ExternalImage. This lets to quickly realize which image is external. Also any further processing on bulk of external image inside TW will be easier.

[jacng]

Hi Mark, Thanks for the reply!!! That sent me down a rabbit hole and I went on to explore a number of things

I didn’t quite get the Incoming.json to work on my system, but anyway took it and ended up with this in my exploration, which I guess is close to your intent

json file attached here if anybody is interested: Bookmarklet - images from Edge Chrome directory TEST 1.json (6.2 KB)

Agreed that the additonal json-stringify by the Import complicates the intermediate json text. For this case, I ended up doing a regex search-and-replace of “[img[” with the replacement pathname in the intermediate json text. But my learning from this is it’s not a good practice in general. Far safer to modify the tiddlers post-import, and easier also, because the fields are readily accessible from the imported tiddlers without having to parse the entire json text.

Other findings that I will udpate in a separate thread. Thanks very much for the reply.