TW5 - keeping track of file movement in pdf/ebook/images catalog app

#1

Hi all,

I assume a common usage of Tiddlywiki will be to catalog and maintain notes of pdf articles, ebooks and image/photo files (in a folder tree). Before I embark on such an application, would like to check how others have done it.

For my purpose, I will be using a Single File TW5 that can be opened on Android phone (using RCX) and Chromebook. My file folders and the Tiddlywiki catalog file will be replicated among my devices.

My stumbling block is I would like to be able to organise my files in the folders (i.e. add, ‘consume’ the files and then delete or move them to permanent folders) without having to manually update the Tiddlywiki catalog file each time. To that end, I envision an external (python) app to scan my file folders and update to the Tiddlywiki catalog file directly (using ‘Tiddlpy’ python lib to read and update Tiddlywiki file) the file movements and file additions/deletions, based on filename. This would likely have to be done outside Tiddlywiki due to browser restrictions to local file access. I have not come across such an application in my search so far.

I came across pretty nice Tiddlywiki cataloging/bibliography applications. Filenames can probably be dragged and dropped into a Tiddlywiki file to create the initial entry, but nothing on scanning folders and updating subsequent file movement/deletion in the folders to a Tiddlywiki catalog file.

Coming back, I would like to get feedbacks on how you use TW5 to catalog and keep notes of your pdf, ebooks and image/photo files (in a folder tree). I’m open to a easier/alternative approach to such an application.

#2

Hi all, there is no response to my earlier post. Please allow me to elaborate a little more and give it another try.

I use a “folder” based approach to process my media files. New files start in the “Download” folder and get moved to progressively more specific folders or deleted as I consumed them. Examples:

  • Newspaper clipping in “Camera” folder → “News” folder → Specific topic folder under “News”
  • Misc images in “Camera” folder → One of several general image folders → Specific topic folder under “Pictures”
  • Pdf/Ebooks → “Download” → “New books” folder → Specific topic folder under “Books”

A favorite file manager (Total Commander on Windows and Android, highly efficient :slight_smile: ) is used to open and manage these media files. Using TW to catalog these media files offers another option to track and manage them with tagging and commenting from the point of creation until they are eventually archived in permanent folders or deleted.

The sticking point to me is file links in TW are broken when these media files are moved to another folder as I processed them. Manual updating in TW will be tedious due to the number and frequency of these file movements. I’m not ready for Node-based TW which i suspect might be better at dealing with it.

i’m exploring a mechanism (outside of TW) to update broken file links in TW file. This may also be used to automatically create tiddler entries for newly added media files and also to update status of tiddlers for deleted/moved files. The approach I shared earlier is to scan the folders, compare them to the contents in TW file and update any file movements back into the TW file directly. The Python “TiddlPy” library can read/write TW file directly.

Another possibility I’m thinking that don’t require altering the TW file directly, is to maintain a dictionary tiddler in one external tid file to store the media file names and their folder locations. TW will reference this external dictionary tiddler (via canonical uri) to derive the folder locations of media files. To update this dictionary tid file, an external program will scan the folders, compare them against the tid file and update any file movement changes into the tid file. New media files will need another mechanism (a plugin to auto-import tiddlers?) to auto-create new tiddlers for the new files in TW.

On the other hand, I believe using TW to catalog media files is a common application of TW, so I could be missing something obvious and overthinking it when there are better ways or existing solutions to do what I thought is needed, for which I seek the feedback from this community, thanks!

#3

I see what you are after however if you use TiddlyWiki to organise files then perhaps you can abandon the separate folders, and just lump them all together in a single folder. Rather than move files between folders, you change a tag on that “file” represented by a tiddler.

  • The files still could be in a range of folders just move them inside tiddlywiki not on the file system.

I suspect there are some equally “upside down” ways at looking at this problem. * For example what if a file is moved tiddlywiki does not know, the first attempt to look at it will show the file is missing and you initiate a search to find its new location.

  • Perhaps just import the whole list of files and folders and build an update method, or a search method, eg tiddler has a fieldname field containing the fieldname and when you want to access the file it first searches the latest file/folder list (in a tiddler) to find where it is located. You could store the “last seen” location.

I can imagine some ways on a desktop to handle this but we always face the nexus between the wiki (in a browser), the file system (especially on mobile) all trying to hide the file structure from you.

Your final solution may prove to be based on a set of compromises influenced by how many files or changes occur, the ease of updates etc…

  • One key here is multiple folders allow duplicate files or filenames - fortunately tiddler titles are unique.
2 Likes
#4

Are you serving the wiki over WebDAV? If so and if you are comfortable getting your hands dirty with some JavaScript, you can write an action widget that you invoke on wiki startup that scans the directory structure and updates/creates tiddlers based on file changes.

If that sounds like something you might want to do, a similar approach is described in my wikifarm writeup and associated videos.

4 Likes
#5

I agree with @TW_Tones on this. Ever since I exclusively used my TW to find information, including references to external files, I found no need to simultaneously maintain a user-friendly directory structure. Principally, everything can go into a single directory (though I’m not taking it quite this far) and any status I want to give a file is done via association with tiddlers, or tags or whatever.

The viability of this of course depends on your usage patterns. This is not a solution that will fit everyone. But sometimes we overcomplicate things by not wanting (or forgetting) to let go of the old ways of doing things…

Have a nice day
Yaisog

2 Likes
#6

Hi all,

Thanks for the thoughtful replies ! The key feedback I receive for such catalog app usage is to put the media files where they are and use TW to track and organise them instead. I will have re-think my system for processing my files. I’m only a few months into TW, would really, really love to be at that stage where I’m comfortable to use TW to track everything. Working my way there :wink:

To @saqimtiaz
Your approach is very interesting, didn’t know that could is possible ! I’m using WebDAV on my phone to serve files but have not make up my mind for my Chromebook desktop setup, I’m a newbie to TW. Will definitely look into that it :slight_smile:

Thanks all !

#7

A different approach, that is client side, allowing you to move files, exists if the following conditions are met:

  • You’ve externalised images (https://tiddlywiki.com/#ExternalImages)
  • All file names always stay the same
  • All file names are unique
  • No non-latin characters in names, including spaces (actually, not sure if this is necessary, but precautionary)

After an image session where you’ve moved files, make a plain path list using whatever tool your platform suppots. e.g.

files/a/b/myuniquefile1.png
files/c/b/myuniquefile2.png
files/b/b/myuniquefile3.png

Then you could copy this list to a tiddler in TW, and run a TW process that would match _canonical_uri by base name and overwrite the current address with the updated address.

#8

TW-Scripts has some solutions for this by @Mark_S, @saqimtiaz and @jeremyruston

See: Scripts in Tiddlywiki — codes, macros, and solutions in TW (kookma.github.io)

#9

I would add if the filenames remain in one place only, and were organised only within tiddlywiki with tiddlywiki metadata, one way to move between “virtual” folders is to use @Mohammad’s Favorites Plugin: Update 4.6.8 - #8 by Mohammad this does not even touch the tiddlers concerned.

1 Like
#10

Love this new term in TW! Virtual folder

#11

Thanks for the tips. Now I have a few options to refresh file links in TW, will check them out.

A side note on “Virtual folder”. My first TW application keep track of web site bookmarks imported from browsers. I discovered that, for my purpose, the folder path in the browser’s bookmarks e.g. “/Software/Note Taking apps/Tiddlywiki/” translate nicely into tags i.e #Software #NoteTakingApps #Tiddlywiki. So a search for the tag #NoteTakingApps in TW now shows all the bookmarks previously under the “/Software/Note Taking apps/” folder tree. Neat.

1 Like