File paths for tw-com edition

Community Projects Documentation
#1

This is an attempt to collaboratively create a $:/config/FileSystemPaths tiddler to control where new tiddlers in the tw5-com edition are saved. @pmario @jeremyruston

Current status:

  • Finished up to and including definitions from this directory: TiddlyWiki5/editions/tw5.com/tiddlers at master · Jermolene/TiddlyWiki5 · GitHub
  • I (pmario) did add some TODOs, to show the directories, which need revisiting
  • Elements higher up in the list will take precedence, if a tiddler has more than 1 tag
    • So the first rule, that produces an output will “take it all”
  • I did sort the directories in the way they are presented by the source code editor!!!
    • So it is less confusing when we create a PR.
  • Saq’s path tool file (updated regularly)
    do not forget the subdirectories! 
[is[draft]addprefix[drafts/]]
[tag[About]addprefix[about/]]

[tag[Commands]addprefix[commands/]]

[tag[Community]addprefix[community/]]
[tag[Articles]addprefix[community/articles/]]
[tag[Community Editions]addprefix[community/editions/]]
[tag[Examples]addprefix[community/examples/]]
[tag[Community Palettes]addprefix[community/palettes/]]
[tag[Community Plugins]addprefix[community/plugins/]]
[tag[Other Resources]addprefix[community/resources/]]
[tag[Community Themes]addprefix[community/themes/]]
[tag[Tutorials]addprefix[community/tutorials/]]
[tag[Meetups]addprefix[community/]]

[tag[Concepts]addprefix[concepts/]]

[tag[Customise TiddlyWiki]addprefix[customising/]]

[tag[Definitions]addprefix[definitions/]]
[tag[table-of-contents-example]addprefix[demonstrations/]]
[tag[Table-of-Contents Demonstrations]addprefix[demonstrations/TableOfContents/]]
[tag[TaskManagementExample]addprefix[demonstrations/Tasks/]]
[tag[Demonstrations]addprefix[demonstrations/]]

TODO check demonstrations directory
[tag[Editions]addprefix[editions/]]

[tag[Features]addprefix[features/]]

[tag[Fields]addprefix[fields/]]

[tag[TiddlyWiki on Node.js]addprefix[fileformats/]]

[tag[Filter Operators]addprefix[filters/]]
[tag[Filters]addprefix[filters/]]
[tag[Filter Syntax]addprefix[filters/syntax/]]
[tag[Operator Examples]addprefix[filters/examples/]]
[tag[shopping]addprefix[filters/examples/]]
TODO -- The following is a new tag. PR has been submitted
[tag[GettingStarted]addprefix[gettingstarted/]]

[tag[HelloThere]addprefix[hellothere/]]
[tag[HelloThumbnail]addprefix[hellothere/thumbnails/]]
[tag[HelloThumbnail-disabled]addprefix[hellothere/thumbnails/]]
[tag[HelloThere Badge]addprefix[hellothere/badges/]] 
TODO check hellothere dir again!

[tag[Hidden Settings]addprefix[hiddensettings/]]

[tag[How to apply custom styles]addprefix[howtos/]]
[tag[Editor toolbar]addprefix[howtos/]]
[tag[Custom Styles FAQ]addprefix[howtos/faq/]]
[tag[Hard Linebreaks in WikiText]addprefix[howto/]]
TODO check `howtos` directory again 

[tag[picture]addprefix[images/]]
TODO check `images` directory again 

[tag[Language]tag[Icon]addprefix[languages/icons]]
[tag[Languages]addprefix[languages/]]

TODO NOTE This has to go before 'Learning'
[tag[KeyboardDrivenInput]addprefix[demonstrations/KeyboardDrivenInput]]

[tag[Learning]addprefix[learning/]]

[tag[Core Macros]addprefix[macros/]]
[tag[Macros]addprefix[macros/]]
[tag[sampletab]addprefix[macros/examples/]]
[tag[Macro Examples]addprefix[macros/examples/]]

[tag[Macro Syntax]addprefix[macros/syntax/]]

TODO check `macros/import` directory again

[tag[Mechanisms]addprefix[mechanisms/]]
[tag[RefreshThrottling]addprefix[mechanisms/]]

TODO check `menubar` dir again

[tag[Messages]addprefix[messages/]]
[tag[Message Examples]addprefix[messages/]]

[tag[TiddlyWiki on Node.js]addprefix[nodejs/]]
[tag[TiddlyWikiFolders]addprefix[nodejs/]]
[tag[SystemTiddlers]addprefix[nodejs/]]

[tag[Platforms]addprefix[platforms/]]

[tag[OfficialPlugins]addprefix[plugins/]]
[tag[PluginMechanism]addprefix[plugins/]]
[tag[Plugins]addprefix[plugins/]]

TODO check `readme` dir again

TODO check `reference` dir again

[tag[ReleaseNotes]addprefix[releasenotes/]]

[tag[AlphaReleaseNotes]addprefix[releasenotes/alpha/]]
[tag[BetaReleaseNotes]addprefix[releasenotes/beta/]]
[tag[Release 5.0.8-beta]addprefix[releasenotes/beta/]]
[tag[Release 5.0.9-beta]addprefix[releasenotes/beta/]]
[tag[Release 5.0.11-beta]addprefix[releasenotes/beta/]]
[tag[Releases]addprefix[releasenotes/]]
[tag[Release Template]addprefix[releasenotes/]]

TODO check the tags in the release notes again. They are not consistent


[tag[Saving]addprefix[saving/]]
TODO chekc `roadmap` again

TODO check the tags in the `saving` dir again. They are not consistent

[tag[Improving TiddlyWiki Documentation]addprefix[styleguide/]]
TODO may be there should be a tag `Styleguide`

TODO check the tags in the `system` dir again. They are not consistent or not there at all

[tag[SystemTags]addprefix[systemtags/]]

[tag[TiddlyDesktop]addprefix[tiddlydesktop/]]

[tag[Variables]addprefix[variables/]]
[tag[Variable Examples]addprefix[variables/examples/]]

TODO check `variables/examples` again .. they have not tags

[tag[Video]addprefix[videos/]]
[tag[Working with the TiddlyWiki5 repository]addprefix[videos/]]
TODO NOTE This Saving with TiddlyFox tag has to go after the videos tag
[tag[Saving with TiddlyFox]addprefix[saving/]]


[tag[RangeWidget]addprefix[widgets/]]
[tag[RadioWidget]addprefix[widgets/]]
[tag[ListWidget]addprefix[widgets/]]
[tag[KeyboardWidget]addprefix[widgets/]]
[tag[Widget Examples]addprefix[widgets/]]
[tag[Widgets]addprefix[widgets/]]

TODO check `widgets` dir again .. some tags are missing or wrong

[tag[Transclusion in WikiText]addprefix[wikitext/]]
[tag[WikiText Examples]addprefix[wikitext/]]
[tag[HTML in WikiText]addprefix[wikitext/]]
[tag[WikiText]addprefix[wikitext/]]

[tag[Working with TiddlyWiki]addprefix[workingwithtw/]]
[tag[WebServer Authentication]addprefix[webserver/]]
[tag[WebServer Guides]addprefix[webserver/]]
[tag[WebServer Parameters]addprefix[webserver/]]
[tag[WebServer]addprefix[webserver/]]
[tag[WebServer API]addprefix[webserver/]]

TODO NOTE These image filters are at the bottom so other filters have a first chance
[regexp:title[(.jpg|.gif|.ico|.png|.jpg|.ascii.tid|.pdf|.svg)$]addprefix[images/]]

TODO check all the tags in the real source editor ... There is more work to do !!

Items for Review

TableOfContents

Almost all the items tagged TableOfContents go to a path that is the title, lowercase, with spaces squished out.

We could create a filter to do the same thing and grandfather in the two exceptions, or have the two exceptions match the pattern. It’s been suggested that spaces should be replaced with hyphens.

TableOfContents

Tiddler Current Path Computed
HelloThere hellothere hellothere
Learning learning learning
Working with TiddlyWiki workingwithtw workingwithtiddlywiki
Customise TiddlyWiki customising customisetiddlywiki
Features features features
Languages languages languages
Editions editions editions
Plugins plugins plugins
Platforms platforms platforms
Reference reference reference
Community community community
About about about
2 Likes
How to test filesystempath easily
#2

I did sort the directories in the way they are presented by the source code editor!!!
So it is less confusing when we create a PR.

#3

It seems to me that you can’t know if this works without testing of some kind.

Is there a script to run this against tiddlywiki. com and create a resulting directory tree? Which can then be compared against the existing tree? Then subtracting out grandfathered tiddlers, see if there are gaps that need to be filled?

#4

The underlaying mechanism works.

We have to have a closer look at the directories, that I did mark with TODO on a per tiddler basis. If the filter is used as it is atm, it won’t work too well.

Not every tiddler in every subdirectory has a tag at the moment. … So the OriginalTiddlerPaths tiddler is used to save them back, where they came from. BUT … It won’t work for new tiddlers, that’s why we need the $:/config/FileSystemPaths

Once we have rules for every existing docs tiddler in the system, we will have a deterministic way to know where new tiddlers will be saved to, depending on their tags or any other field, we may define.

This will enable us to improve the mechanism described in: Making changes to tiddlywiki.com via GitHub and Pull Requests (PR)

#5

I’m not sure what we’re supposed to do with these. For instance, here’s all the tags used by tiddlers currently in the howto directory:

Concepts
Custom data-styles
Customise TiddlyWiki
data-tags-styles
Editor toolbar
example-hardlinebreaks
example-test
FAQ-group
Features
Formatting text in TiddlyWiki
Hard Linebreaks in WikiText
How to apply custom styles
Learning
Saving
TiddlyWiki on Node.js
Windows
Working with TiddlyWiki

They seem all over the board. What rule would be added? Interestingly, none of them had the tag ‘Howto’ .

1 Like
#6

If we need to tag a few tiddlers or even change their locations while we set this up, that should be acceptable as a one time thing.

In fact, some tiddlers are definitely in the wrong directories at present. A lot of the clutter is probably the result of people writing documentation and guessing at where to put the files in the directory structure.

#7

That’s why I’m trying to understand the nature of the task. Is it about finding the perfect filter, or about fixing the data. Or ?? Oh, I probably should have mentioned one item didn’t have tags at all.

About renaming files. Renaming is deleting and recreating in another directory, right? Is git smart enough to maintain the history for a file when this happens?

After applying the filters that come before howto, (well actually the inverse of each tag), there are 38 tiddlers with no reason to be in the howto directory per se.

Should a “howto” tag be added to all of them, or new rules to sort them, or ?

I note that several of them say “Custom …” raising the question whether they should have been in a customising dir instead.

Adding a Twitter Follow button
Adopt a Titles Policy
Attribute Selectors
Concatenating text and variables using macro substitution
Creating SubStories
Custom data-styles
Custom styles by data-tags
Custom styles by data-tiddler-title
Custom styles by user-class
Custom Styles FAQ
Editing Tiddlers with Emacs
Editing Tiddlers with Vim
Formatting List Results as Tables (no CSS)
Formatting List Results as Tables with CSS - Specified Columns Methods
Formatting List Results as Tables with CSS - Variable Column Method
Hard Linebreaks with CSS
Hard Linebreaks with CSS - Example
Height of text editor
How to add a banner for GitHub contributions
How to apply custom styles by tag
How to change the sort order of sub-branches in a TOC macro
How to create dynamic editor toolbar buttons
How to Customize TiddlyDesktop
How to embed PDF and other documents
How to export tiddlers
How to hide the author’s and other fields with CSS
How to put the last modification date in a banner
Insert link
Insert picture
Making curved text with SVG
More actions
Sharing a TiddlyWiki on Dropbox
Simple ways to write protect tiddlers
Structuring TiddlyWiki
Text preview
The First Rule of Using TiddlyWiki
Using Excise
Using Stamp
Windows HTA Hack

1 Like
#8

I think what we want to do is as much as possible describe the existing structure with filters. Where that breaks down we fix the data to allow a filter to be used. The idea being to minimize data changes.

In my experience, as long as you don’t also modify the contents of the files when you move them, the history is retained.
If you need to say move a tiddler and add a tag, make that two separate commits.

I think we need to compare the kind of information in these tiddlers vs the ones currently in the customising directory to see if they would fit under that same umbrella. A quick looks suggests that a lot of them are indeed very similar in nature (“Custom styles by *” for example, whereas “Editing Tiddlers with Emacs” and the like are not about customising TW. I wonder if a lot of those might fit under the “Learning” tag.)

#9

The ‘customizing’ ones were tagged ‘How to apply custom styles’. Going with the principle of “Change the filter first”, I’ve added the tag rule.

#10

First rule of filter club: Change filter if possible
Second rule of filter club: Preserve current tags over tag changes. i.e. moving files is “harmless” as far as the end user is concerned whereas by now whole systems are based on tags

I propose that all tiddlers in ‘howto’ currently tagged as ‘Learning’ be moved to the ‘learning’ directory.

Edit: I propose that any tiddlers in ‘howto’ that match existing filters be moved to the respective directories. That should take care of more than half the discrepancies.

#11

I think it might be helpful to start a branch that actually implements the changes that are needed and to use that to collaborate around this. Unless someone beats me to it, I can hopefully find time to do so next week.

@jeremyruston could you please confirm whether moving the location of some tiddlers in the directory structure so that they are better organized in a logical manner, and the paths can better be described with filters, would be acceptable?

Such changes would naturally be kept to a minimum and would not impact users of tiddlywiki.com

#12

Yes absolutely. I would be delighted to get that sorted out. We definitely shouldn’t let the inconsistencies of the existing layout slow us 02 down.

#13

Note that whenever we add a path prefix, it needs to end with a trailing slash.

#14

I have thrown together a quick hack that shows:

Tweak and adjust filters in pathFilters

#15

For ‘Tiddlers whose paths are different…’ it was giving about 10 to one false positives because of the various creative file namings. Since we only need to match the paths, the modified code below seems to achieve the objective.

So, what is the :cascade run expression?

My thought was to use list code like yours to build a shell script that could then be used to move all the files.

\define cascadeFilter() [<currentTiddler>] :cascade[{pathFilters}splitregexp[\n]] +[!is[blank]]

\define sanitize() <|>|~|\:|\"|\||\?|\*|\^

// this isn't foolproof in how it handles encoding special characters in the tiddler title, so there might be some false positives //

<$list filter="[all[tiddlers]!is[system]filter<cascadeFilter>]">
<$let oldPath={{{ [[$:/config/OriginalTiddlerPaths]getindex<currentTiddler>removeprefix[../../tw5.com/tiddlers/]split[/]butlast[]join[/]] }}}
			newPath={{{ [<currentTiddler>] :cascade[{pathFilters}splitregexp[\n]] +[split[/]butlast[]join[/]] }}}>
			<$list filter="[<oldPath>!match<newPath>]" variable="null">
			
			* Tiddler: <$link/>
><pre>old path: <$text text=<<oldPath>>/>
new path: <$text text=<<newPath>>/></pre>
			
			</$list>
</$let>
</$list>
1 Like
#16

Thanks Mark. That occurred to me regarding file names but I was out of time to work on this today.

I have added your code as a separate tiddler with a few improvements to handle titles with slash characters: Tiddlers whose paths are different with filters applied - path only

Regarding filenames, I think the issue is going to be that when one of those tiddlers gets edited via node.js in the future, the file name will change if the tiddler matches one of those filters. I am not sure to what extent we try to resolve that versus deal with it on a case by case basis when it comes up.

I think the bigger priority needs to be that new tiddlers with the appropriate tags land in the correct place.

The :cascade filter run prefix implements the same logic as used in the file path filters in a filter run. It hasn’t landed in the pre-release just yet so I added it manually so we could use it for testing.

For more details see: https://github.com/Jermolene/TiddlyWiki5/pull/6168

Once we have all the changes lined up we should indeed be able to output a bash script from wikitext, to move the files to their new locations.

#17

Which list did you use for your TW page?

I note that the “saving” tag is in the wiki list here (Discourse) but not in your path filter. I can’t figure out how that would happen unless you were using an older list or unless Pmario (or you) updated in the last 37 minutes.

I’m assuming that the Discourse filter list is the one that should be canonical? So I could take it, process it, and put in your TW page to see how things are looking?

Thanks!

#18

I did make a bunch of updates to the list both on Discourse and on the TW, if you refresh both they should be in sync, except for the comments that have been removed in the TW copy.

What is making it difficult to keep them in sync is the extra comments in the list here in Discourse. If we remove them to somewhere else, that will make it easier to ensure the Discourse filter list is canonical and we can easily test it on the TW.

I am about to turn in for the night, but some easy gains are to add rules for the following tags:

  • WebServer Guides
  • WebServer Authentication
  • Variable Examples

Edit: the change history of the wiki post is handy in these situations too.

#19

If the comments are formatted like:

#TODO Fix broken tag names

Then they can be removed by prefix. It seems like they might be useful. Or if we just have a convention where any comments start with TODO or NOTE that would work too.

Requesting some tiddler tag fixes

#20

I might not get to it before early next week, but I will set up the testing TW for paths to auto refresh with the latest tw-com content.

For now I have updated the pathFilters on the test wiki.