Internal names of tiddlers in node.js tw

[jypre]

I’m currently in the process of converting my wikis to node.js in order to get access to tiddlers as single files. I’ve just discovered that if the “foo” tiddler is typed as javascript, then it is saved as “foo.js” and “foo.meta” which is just I was hoping for! Same for css or json. There is no such separation for regular tiddlywiki contents, though. Shouldn’t it be also the case, even if only if indicated so in user preferences?

I’ve also had a few experiments with name collisions. I saw that, for instance, “foo/bar” and “foo:bar” all wanted to be named “foo_bar” and so one will be named “foo_bar_1”. The logic is correct but I would have preferred the alternatives names start at 2 instead of 1 out of logic and also because “_1” looks terribly like “_i” or “_l” depending of your current font. An other point I saw is that “$:/user/foo” would be named “$__user_foo”. That “$” in a filename is a pain on linux for further scripting with generated file name because it’s used for accessing shell variables values. Having $ for naming system tiddler is great but not for the inner name.

Naming convention could be many. For instance:
#1: Thanks to the mechanism for avoiding name collisions, we could replace $ with @ which has no special purpose within a shell, without worrying.
#2: avoid spaces and any punctuation, even points, replacing them with underscores. and any sequence of several underscores would be reduced to just one.
#3: only allow a restricted set of accentuated letters and replace others by underscore. or you could replace tolerated accentuted letters by their non accentuated letter (even if this change prounouciation of course. for instance “coup d’état” could translate into “coup_d_état” or “coup_d_etat” or “coup_d_tat”.
#4: and many more possibilities.

Thus the naming convention would be better as a customizable mechanism with a default implementation in the core and others available as plugins. Pehaps each possibility could be an option and one could select which options he want in order to get the customization it wants. plugins would offers one of several such options.

[jeremyruston]

Hi @jypre

The filenames used to save tiddlers under Node.js can be heavily customised:

https://tiddlywiki.com/#Customising%20Tiddler%20File%20Naming

The only thing that can’t be customised at present is the counting scheme for collision avoidance.

[jypre]

Thank you @jeremyruston I’m looking into that. I states the point about a special cases for tw5 tiddlers being a single .tid and not possibly anything else. why not a .meta for them too, leaving the .tid as purely the text of it? It would surely be better to have a .tw5 extension for it to avoid confusion with present .tid.

Otherwise, it is really great. I would say that as usual, tiddlywiki allow an extreme customization, but you may not know about the possibility or how to do it. But the present forum is also here for that!

[TW_Tones]

@jypre I have in fact proposed a few times that single file tiddlywikis optionally use the .tw5 or .tw extension so they can be shared in emails more readily since html scares a lot of email clients. Perhaps something to reflect its data eg .twd or its field .twtext?

I have an association on my computer which treats .tw files as html and opens in the browser, most savers worked with it at one point.

[jypre]

My idea was of splitting a tiddler file fubar.tid into two files:

• fubar.meta for the metadata, that is the header at the beginning of fubar.tid ;
• fubar.tw5 for the text which is al the remainder of fubr.tid

The advantages seen by me were:

• as I would put these files into a repository such as git, I would clearly separate the two kinds of data, being clearer and eventually much shorter when treating only the metadata;
• easier to edit the text with an editor oustide the browser;
• easier to run various utilities on the files (be it for headers or corpus).

One thing that has come to my mind relative to git and such toolss is the renaming problem. Normally, as I rename tiddler “foobar” as “fubar” I should inform git that “foobar.meta” has become “fubar.meta” and same idea for “foobar.tw5”. But I must do it out of the browser, by hand, before doing it on tiddlywiki otherwise all this information would never be made available.

Using git immediatly as a saving method from tiddlywiki is not such a good idea as it would add lots of noise in the history of the git repository and you’re also suppose to document each update in a meaningful way. So the problem is not so simple. I don’t see any automated way to solve the problem. IMHO’ any renaming should be handed as a separated batch of operation done on the directories with git which would correspond to a dedicated git transaction as well, immediately pushed.

But forgetting may happen and will happen with time. Surely it is/could be written on logs so that it would help making the things correctly? Perhaps it could be noted that a renaming has been done with the GUI and create a special file, not a tiddler, indicating what has happened. Each VCS would offer different ways to handle that information by its user.

Renaming must not be forbidden because as long as the tiddler has never been on the VCS system, it’s OK to rename it and a typo at this stage should be fixed quickly and simply.

[jypre]

I tried some filters in $:/config/FileSystemPaths

Basically, I want to have only a-z - _ . in names, plus / in the path part of the name.

I also want to replace things liki éèêë with e. This is done by default but it seems I need to do it if I want to get rid of unwanted chars (especially spaces) because that replacement comes too late.

So here the content of $:/config/FileSystemPaths:

[prefix[$:/user/]!has[draft.of]removeprefix[$:/]addprefix[_system/]lowercase[]search-replace:g:regexp[é|è|ê|ë],[E]dump:v[before unfit]search-replace:g:regexp,[-]dump[fit]]
[is[draft]search-replace:g:regexp[/|\\],[_]addprefix[drafts/]]

I defined in a separated tiddler the unfit macro as

\define unfit() [^-a-z0-9_/.]+

And I have a test tiddler named $:/user/étiré/éphémère « à l'œil ». Ça va ? but when I update the $:/config/FileSystemPaths (no change) the server has a fatal error:

<empty string> syncer-browser-tiddlyweb: Dispatching 'save' task: $:/StoryList <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Checking dirty status <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Dispatching 'save' task: $:/config/FileSystemPaths <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Checking dirty status <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Dispatching 'delete' task: Draft of '$:/config/FileSystemPaths' <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb-connection: <h2>''Network Error''</h2>It looks like the connection to the server has been lost. This may indicate a problem with your network connection. Please attempt to restore network connectivity before continuing.<br><br>''Any unsaved changes will be automatically synchronised when connectivity is restored''. <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Sync error while processing delete of 'Draft of '$:/config/FileSystemPaths'': XMLHttpRequest error code: 0 <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Checking dirty status <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Dispatching 'delete' task: Draft of '$:/config/FileSystemPaths' <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb-connection: <h2>''Network Error''</h2>It looks like the connection to the server has been lost. This may indicate a problem with your network connection. Please attempt to restore network connectivity before continuing.<br><br>''Any unsaved changes will be automatically synchronised when connectivity is restored''. <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Sync error while processing delete of 'Draft of '$:/config/FileSystemPaths'': XMLHttpRequest error code: 0 <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Checking dirty status <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Dispatching 'delete' task: Draft of '$:/config/FileSystemPaths' <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb-connection: <h2>''Network Error''</h2>It looks like the connection to the server has been lost. This may indicate a problem with your network connection. Please attempt to restore network connectivity before continuing.<br><br>''Any unsaved changes will be automatically synchronised when connectivity is restored''. <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Sync error while processing delete of 'Draft of '$:/config/FileSystemPaths'': XMLHttpRequest error code: 0 <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Checking dirty status <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Dispatching 'delete' task: Draft of '$:/config/FileSystemPaths' <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb-connection: <h2>''Network Error''</h2>It looks like the connection to the server has been lost. This may indicate a problem with your network connection. Please attempt to restore network connectivity before continuing.<br><br>''Any unsaved changes will be automatically synchronised when connectivity is restored''. <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Sync error while processing delete of 'Draft of '$:/config/FileSystemPaths'': XMLHttpRequest error code: 0 <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Checking dirty status <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Dispatching 'delete' task: Draft of '$:/config/FileSystemPaths' <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb-connection: <h2>''Network Error''</h2>It looks like the connection to the server has been lost. This may indicate a problem with your network connection. Please attempt to restore network connectivity before continuing.<br><br>''Any unsaved changes will be automatically synchronised when connectivity is restored''. <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Sync error while processing delete of 'Draft of '$:/config/FileSystemPaths'': XMLHttpRequest error code: 0 <empty string> [$:/core/modules/utils/logger.js:52:26](chrome://devtools/content/webconsole/$:/core/modules/utils/logger.js)

<empty string> syncer-browser-tiddlyweb: Checking dirty status <empty string>

a

and it keeps doing.

There are only 3 tiddlers on displaY:

• $:/config/FileSystemPaths
• unfit
• $:/user/étiré/éphémère « à l'œil ». Ça va ?

If I remove the filter run of unfit (after restarting the server)’ things get back OK. Any idea of what causes the problem?

[joshuafontany]

The filesystem adaptor does not have access to any global macros, as it runs outside of the (fake)document context on the server. Also, the console log from the browser only shows the connection failed (I suppose when the server crashed). You would need the server log to determine the file the error occurred in, line number, etc.

[TW_Tones]

I cant answer your question, but have you considered making use of the following?

• ## slugify Operator
  
• ## duplicateslugs Operator
  
• possibly 5.2.0 Prerelease ## unusedtitle Macro
  

[jypre]

Here is the terminal log. The first line is that of my dump filter, showing that if I can’t use a maco, I can use my own filters (coded in js within a tiddler, as usual):

«before unfit» 0 tiddler []
$:/core/modules/filters.js:244
						operand.value = widget.getVariable(operand.text,{defaultValue: ""});
						                       ^

TypeError: Cannot read properties of null (reading 'getVariable')
    at $:/core/modules/filters.js:244:30
    at Object.$tw.utils.each (/usr/local/lib/node_modules/tiddlywiki/boot/boot.js:126:12)
    at $:/core/modules/filters.js:240:15
    at Object.$tw.utils.each (/usr/local/lib/node_modules/tiddlywiki/boot/boot.js:126:12)
    at operationSubFunction ($:/core/modules/filters.js:229:14)
    at $:/core/modules/filterrunprefixes/or.js:20:19
    at $:/core/modules/filters.js:316:4
    at Object.$tw.utils.each (/usr/local/lib/node_modules/tiddlywiki/boot/boot.js:126:12)
    at $tw.Wiki.filterFunction ($:/core/modules/filters.js:315:13)
    at $tw.Wiki.exports.filterTiddlers ($:/core/modules/filters.js:200:12)

In $:/core/modules/filters.js:244:30 we have operator.operator which fails because the operator variable is null:

			$tw.utils.each(operation.operators,function(operator) {
				var operands = [],
					operatorFunction;
				if(!operator.operator) {

Perhaps we could code the passage like this:

                            if (!operator) {
                                    operatorFunction = nop
			} else if(!operator.operator) {
				operatorFunction = filterOperators.title;
			} else if(!filterOperators[operator.operator]) {
				operatorFunction = filterOperators.field;
			} else {
				operatorFunction = filterOperators[operator.operator];
			}

The nop operator is an operator doing nothing, which would be declared somewere or here as an anonymous function. A bit of logging could be added to help the developper spot the problem. I am not yet able enough to make the proposal now.

The important things to me is to know I can do my own filter to be used here without any problem as it voids my being blocked. I shall report what I’ll get to do to acieve this.

[jypre]

@TW_Tones: Thak you! slugify is indeed most of what I wanted to achieve and is thi cause of the comportment I had noticed in some circumstances when I tried to write my filter.

As for the slugify doc, I think it would be better to code for the slug name not as ‘‘bold’’ but either as //italics// or as code case or ‘between quotes’ or “between double quotes” (typographic concerns). I had been looking for what a kind of field a “slug field” and I found nothing. And then only did I realized that “slug” was the name of the field. The fault being that of an error in typographic conventions. OK, I can propose the change now that I (can) have a node.js instance of the full edition. But the docs don’t really say where to go for this kind of things.

[jypre]

Here is my filter for my naming convention, done possible because of your explanations:

[prefix[$:/.]!has[draft.of]removeprefix[$:/]search-replace:g[_],[-]search-replace:g:regexp[/+],[_]slugify[]dump:v[slugify]search-replace:g[_],[/]search-replace:g:regexp[^\.+(-*|\.*)*|\.+],[.]search-replace:g:regexp[-\.|\.-],[-]search-replace:g:regexp[-+],[-]search-replace:g:regexp[^-|^\.|-$|\.$],[]addprefix[_system/]]

And with my dump operator (previously given on the GG forum) to see what steps it notably has:

[prefix[$:/.]!has[draft.of]dump[raw]removeprefix[$:/]search-replace:g[_],[-]search-replace:g:regexp[/+],[_]dump[prepared]slugify[]dump:v[slugify]search-replace:g[_],[/]dump[slash]search-replace:g:regexp[^\.+(-*|\.*)*|\.+],[.]dump[dot init]search-replace:g:regexp[-\.|\.-],[-]search-replace:g:regexp[-+],[-]dump[before ends]search-replace:g:regexp[^-|^\.|-$|\.$],[]dump[corrected]addprefix[_system/]]

With it, the horrible title of a tiddler which is

$:/.-.. -- .user/étiré/éphé_mère. « à … l'œil. . . ? ». Ça… "va"... ? 'daß' teþ ^i~%°C ij †.-v4.16.108

is translated into the cute

_system/ser/etire/ephe-mere-a-loeil-ca-va-da-te-ic-ij-v4.16.108

(I have not tried to correct the problems with some characters previously noted.)

One last question: if my tiddler foo is of type dictionary, it is translated as foo and foo.meta. It would be better to have foo.dic instead of foo. This should be done within $:/config/FileSystemExtensionsbut how? The examples give use of the tag operator, but surely I could use the determination of the type field instead?
So I thought and I use as a content of $:/config/FileSystemExtensions:

[field:type[application/x-tiddler-dictionary]then[.dic]]

and in effect the dictionary foo is now translated in foo.dic and foo.dic.meta? Why this latter name and not rather foo.meta?

[jypre]

Just a quick note to say it is possible to have an extension for tw5 tiddlers (expressely typed as tw5) thanks to the following $:/config/FileSystemExtensions tiddler:

[field:type[text/vnd.tiddlywiki]then[.tw5]]

with that, tw5 tiddler foo will be saved as foo.tw5 for the text and foo.tw5.meta for the header.