Idea of multi field data tiddler like a data base

#1

Hello Fans and Developers of TiddlyWiki,

I am using TiddlyWiki mainly as a Wiki for several topics for private, and hopefully in the future also for my own business.
During my last session of providing information I got the following idea for a function I miss and I want to discuss here.

What do you think about a function to have the possibility to create data-tiddlers with multiple fields and a selection method to read them out?

Here an example for my idea (it could also be a list of contacts or similar):

title: Box_content
type: application/x-tiddler-dictionary

Box_Label:Content:Box_Color:Location
1:Brush:Red:Garage_1
1:Soap:Red:Garage_1
Alpha:Brush:Blue:Kitchen
Alpha:Pen:Green:Garden
Rain:Umbrella:Black:Car
Fall:Fork:Red:Garden

And finally a selection command like $list

<$select from Box_content Content,Location where Box_Color="Red">

What will return

Brush Garage_1
Soap Garage_1
Fork Garden

Of course this is only an idea but maybe others are also interested and maybe this idea will be developed as a plugin or integrated in TiddlyWiki.

Stefan

1 Like
Announcing: TiddlyWiki for SQL users
#2

There are 2 main problems with this type of data tiddler

  1. refresh handling
  2. performance

add 1)
In TW the page is refreshed whenever any tiddler is changed in any way. So the UI “state” can be automatically updated.

The refresh mechanism is optimized. Every widget re-calculates its internal state only if the changed tiddler is actually “connected” to that widget. So the refresh can be fast.

The problem with a data tiddler is, that it can hold many different “states”. In your example you define 7 different states. So if the line “Fall:Fork:…” changes all the other widgets, which may read other lines in the data tiddler would need to read the info and re-calculate their state, even if it did not change at all.

add 2)
That would be a big performance hit, the larger the “database” gets. So data-tiddlers are great for configuration data, or data that rarely changes. As soon as you want to write to them, they potentially cause a performance problem.

The problem for core developers is, that users tend to use existing functions in ways we never ever did anticipate. → That’s a good thing but we always have to make compromises.

hope that makes sense
-mario

#3

Hi @pmario ,

thank you for your answer.
If I understand it right I agree that my idea cannot be realized by the existing functionality of data tiddlers. It need a new type of tiddler to avoid this refresh procedure. Only on request the content should be reloaded.

Stefan

#4

You can have a more complex data structure (more than key:value pairs as in tiddler dictionary) using JSON data tiddler, so for your example something like

title: Box_content
type: application/json

[
  {
    "Label": "1",
    "Content": "Brush",
    "Color": "Red",
    "Location": "Garage"
  },
...
]

There are filter operators for handling JSON data.
I use this approach for some small configuration data that would be unwieldy to keep in separate tiddlers.

I think it could be a handy idea to support some CSV or similar data structure with appropriate filter operators to be able to do these SQL-like queries. CSV would be much easier to edit or read in raw form, but:

  • it would probably require a substantial effort to implement
  • it would have the exact same performance issues as JSON or dictionaries
  • for very large datasets and complex operations, as I understand, the better solution is to keep each data point as tiddler with dedicated fields anyway

So I’m not sure it would make much sense to implement it at all.

1 Like
#5

Hi @vilc,

Do you think that external CSV files should also be supported? That would of course be an extension of my idea, which was aimed more at simple filtering and selection of the data.

Stefan

#6

Perhaps it’s a nice idea for a plugin, but the more I think about it, the less sure I am it is even a practical idea, it would probably be very niche.

Big and complex data will be best served in the form of individual tiddlers. If we explicitly want to work with big CSV or similarly structured datasets, I’m not sure if TW is the right software to attempt that.

For small and rather static datasets, e.g. configurations, there is not that much to be gained by using yet another format (next to dictionary tiddlers and JSON tiddlers), and it adds complexity.

1 Like
#7

Hi @vilc,

I think you are right.
My (individual) use case is not more than 500 lines with not more than 10 fields.
he JSON tiddlers may provide the function but I think I will lose the overview quickly if I have to use this format.
A simple table structure is much easier to use.

Stefan

#8

The Multi-Wiki-Server plugin will provide TiddlyWiki clients access to large media files - it also provides the framework for JS developers to interface to SQLite databases. This would allow TiddlyWiki clients relational access to small and large volumes of data stored server-side.

By quering the databases for only the data requested, wrapping the data in tiddler(s) (if not already stored as a tiddler), and sending to the client solves the problems with the refresh mechanism and performance issues client-side; which @pmario so rightfully describes in the prior post Idea of multi field data tiddler like a data base - #2 by pmario.

However, still months away :frowning: - but the future looks so bright - kitty will need shades.

3 Likes