Allow control of fieldnames included in JSON and CSV exports

Developers
#1

The macro $:/core/modules/macros/jsontiddlers.js formats tiddlers as JSON its filter parameter relates to the titles of tiddlers to export.

  • csvtiddlers also does something similar.

Selective export of fieldnames

I am wondering about extending this macro jsontiddlers and that for csvtiddlers to allow selective export of fieldnames, perhaps a field filter that may include +fieldname and -fieldname so we can selectivly export fields. For example we may want to export only a list of titles and their color field.

  • I will also look at a non-destructive import that would apply the color to tiddlers without a color field and optionaly create missing tiddlers. This would allow the import of selective fieldname exports.

Export to tiddler not only download file

At the same time I see value in the export of tiddlers to a file (with or without selective fields being chosen) written to a tiddler / tiddlers text field. ie JSON tiddlers saved in a tiddler called tiddlers.json because one can often bypass the download and upload step and just drag and drop such a tiddler to the target tiddler.

  • permitting this would allow one to export tiddlers to a tiddler for archive or even conversion to a plugin, all in wiki and no file handling needed.

As it stands the jsontiddlers.js and csvtiddlers.js macros internaly handle the fieldnames exported (all), and will need modification if the Export to tiddler not only download file is going to do the same.

  • I will handle that in a different topic along with a modified import option.
  • That is look at where to make changes for export to also save to a tiddler.

Your opinion or support

Can you assist in identifying how to allow selective field exports by modifying jsontiddlers.js and csvtiddlers.js to accept a fieldname filter (a named variable) similar to exportFilter.

  • Or shall I take the approach of an export template like $:/core/templates/plain-text-tiddler

I can explain the why if asked, but I suspect you may see the value.

Is it possible to "archive" tiddlers in a single file wiki?
#2

This sounds like a very good idea, with the default just being the subfilter fields[]. It might be challenging to get the interface exactly right, because an arbitrary filter seems essential, but I also like your -fieldname / +fieldname possibilities. Hmm, I believe that requires more thinking…

Another good idea. The UI for this will need to be crystal-clear, though. We wouldn’t want to accidentally overlay when we want to merge, nor vice-versa.

A third wonderful idea! I have plenty of temporary files laying around that serve no purpose after I drag them to another wiki. I think it would makes some sense to put them in the $:/temp namespace in the source wiki.

1 Like
#3

This is an idea that has a broader connection to merging tiddlers on import!

I think I raised questions about that here once, when I was wrestling with two overlapping archival resources that had different strengths (each had certain up-to-date contents for fields that were missing, or with outdated values, on the other). I also wanted a capacity to batch-check (or batch-uncheck) exactly those import-candidates that would meet a certain filter condition, including but not limited to “would overwrite an existing tiddler (pink row)”… And I think the reply at the time was something like “tw’s core import interface is really complicated and eventually needs an overhaul, but in the near future it’s brittle and difficult to embrace feature-development for imports”.

At any rate, I think this idea might deserve a separate thread, because variations on this theme — non-destructive and merge-condition imports — is something this thread title (focused on exports) would not seem to cover.

#4

Yes, merging tiddlers is one way to look at what I came to as configuration management. For example I have a system tags tool, that allows one to find and use system tags that are not yet used. It looks even better when the tags are coloured to group them. I was looking for a way to transfer the colors without replacing tags. This of course has been generalised during my research both to use existing code and tools for more use cases.

No actualy stick with this thread, because a selective export is also a way of preparing and selective import.

#5

Well this sounds like a perfect occasion for working with cascade conditions without needing to create an actual tiddler!

There are two steps already taken, in my wikis, toward making tag-pills carry helpful info without relying on having a tiddler:

Why not add: cascade conditions for color and style, where having a color field is just one way (albeit the “top dog” way) to get color applied to tags, based on conditions!

If the tag style is applied via cascade, lots of things are possible:

  • Have a certain color apply to all system tags, and/or tags that start with a certain string
  • Use palette colors and other smart color adjustments to make certain classes of tag stand out well and fit your wiki’s style even across dark/light palette changes.
  • Have a color or style for tags that are “grandparent” tags (tags which not only have children, but which have children with children)… (Maybe this could be a dimension of style that doesn’t conflict with others, such as double-outline)
  • If there is a tiddler, have a style apply if that tag’s home tiddler has a list field! (Maybe this could be a dimension of style that doesn’t conflict with others, such as underline)
  • No need to create an actual tiddler with a color field just to get the tag-pill colored. But color field still works, like a manual override.
  • No worries about overwriting tiddler content just in order to get desired tag style!
1 Like
#6

@Springer I think what you are suggesting is a great approach for the example I gave transferring color to tags, however the example was given to illustrate a mechanisium that could be used for other use cases. So perhaps I need to change my mind and suggest a new topic to manage and transfer colors through a color cascade mechanisium as you propose.

The key mechanisums I am proposing in this topic are;

  • the ability to selectively include fields in the export
  • the ability to selectively include fields during imports
    • Apply field changes rather replace tiddlers
  • Enhancing the export process (Import for that matter) to allow export output to tiddler within the current wiki.

Part of my design approach is to leverage existing tiddlywiki core to keep code and interface changes as simple as possible, with a view to adding to core one day.

Back to your color suggestion;
Color cascade using;

  • color field
  • look up title from a list of colours indexed by title
  • calculated color
  • etc…

However it seems to me this could be further generalised to a cascade that effectivly delivers virtual values for fields from a cascade and or a lookup or actual field value.

  • Imagin <<get-field color>> (a function) being used in tag pill colors which uses a function that is a cascade.
#7

Yes, I’d be excited about all of these — especially the import-related ideas.

The reason I suggested an additional thread for the import issue is just that people might look at thread titles to decide what to read, and folks who need selective (non-destuctive, or merge-oriented) import — without having export-specific needs — might not realize that this thread should interest them. (Or maybe it work help just to edit the title to something like “Add field-specific controls for export, import, bundling” …?)

(And maybe we should even coin another word for the… “bundling?” process that is like export, but wrapping the contents into a tiddler that is very much still in the wiki, so… not really an export, right?)

#8

Reasons I see them as intimately related is because;

  • a filtered export, once imported can be identical in outcome to a filtered import.
  • If one wants to merge an import with the current wiki this would work with any import weather it was selectivly exported or not, however if the intention is to merge, it would be helpful to imply this when exporting so there is no accidental “full” import when it was intended as a merge.
  • The idea of writting what would be an export to file to export to tiddler touches the same import and export code.

Yes we do need to establish some appropriate words. perhaps

  • Filtered imports (new) and exports (already) where a filter can be used to choose what within an importing file, what will actualy be imported. To reach into field values would be helpful.
  • Merge on imports (selective import and merge with existing tiddlers)
  • “Import and export to tiddler” rather than file should be a toggle to enhance existing import and export mechanisiums. Eg;
    • on import you can rename the import tiddler, set it to type plugin and you end up with a plugin
    • On export a JSON file sent to a tiddler, set it to type plugin and you end up with a plugin

I am not against this just think it may be premature until the solutions are determined. Are they part of one or more solutions or one grand set of selective import/export and merge tools.