Rapid Database development tools proposal

#1

Background:

I have long felt tiddlywiki out of the box is very close to a quick way to build one or more databases. For example a contact manager, a Todo list etc…

Simply creating tiddlers and adding fields goes a long way.

However as one builds a solution a range of desirable features would help on this journey for example;

  • Being able to define field formats, to test validity or use a custom edit eg select from list or existing.
    • This also brings into view the use of field definitions and field types
      • many of these can be reused
      • there is actually a somewhat limited number of possible field-types and their display and editor tools, so why not provision them upfront?
  • Being able to view fields arranged into a form
    • Be able to change that form into an editable form (not using the tiddler editor)
  • Be able to manage a per-tiddler and wiki wide modes, such as read only, editor or designer
  • Provide tools so that if you add a field to a particular tiddler type then this fields is available to all tiddlers of the same type
    • This allows you to progressively build your database
  • A set of tools to easily create new tiddlers of a given type to add new items. Templating.
  • Then it is easy to make use of all the other tools, features, plugins against such a database.

Now I have done this a multitude of times, and it gets quite complex as the features are added. It becomes too complex to document and publish and so often bespoke decisions are made.

Proposal

I would like to Assemble a team to develop such a solution that may eventually become at least a core plugin. However we have not found a very good way to develop such teams in the community, except perhaps within the core developers.

Key goals

Provide a set of tools that simplifies building database like solutions on top of TiddlyWiki such that it does not need a database administrator, its adaptive and maximally automated whist keeping it to native tiddlywiki as much as possible.

This approach will include;

  • Support for defining fields and having field tiddlers (like tag tiddlers)
    • each field can have one of a set of field types and new ones can be created
  • Support for defining field types and having field type tiddlers (like tag tiddlers)
    • allowing input format, value selection etc… for each field type
  • Simple form or tiddler type management - field order
  • Simple tiddler and or wiki modes

Because I have being down this path so many times I would like to lead this project setting goals and directions towards its development and perhaps in time bring on some co-leaders.

If you are interested please let me know and share any methods you are aware of that may assist. For example;

  • It is possible to reference fields via a procedure so that it can locate the field definition, look up the field type, determine if it is read only or view mode and present the field as needed
  • It is simple to include a list of fieldnames that need to appear on a form and drag and drop reorder them.
    • Especially if applied for a set of similar tiddlers eg a contact tiddler
  • When in a designer mode it is trivial to include a button near any field on a form so the designer can define the field, select the field type and encode the preferred edit/display method.
  • The relink plugin is a friend in this kind of solution.

Further clarification about this topic is posted here Rapid Database development tools proposal - #16 by TW_Tones

1 Like
#2

See: Customizing EditTemplate field rendering

The Field-Editor cascade can do that.

The Story Tiddler cascade can do that. The EditTemplate is nothing more than a form named EditTemplate. If a tiddler has a “draft.of” field it will be shown.

If a tiddler is not a draft the ViewTemplate will be shown. That also a predefined form. You can define as many forms as you want.

Currently TW does not have a concept of “field inheritance”, but a custom-form field could be used to create “templated” tiddlers as described above. Using the Field-cascade and the StoryTiddler cascade to switch between forms, that depend on the custom-form field. The implementation would be a template.

It is not recommended to create a new “type” as eg: text/vnd-tiddlywiki-something since doing so will need core java-script support.

If it is not possible to be documented, then it can not be published. – So my suggestion would be to do the documentation first in plain text, without code in mind.

This would be called a specification. Only if the specification makes sense and is free of discrepancies it can be implemented in a consistent way. But the code itself may be the last step.

I think all the mechanisms needed are there.

Hope that makes sense.
-m

#3

@pmario thanks for your contribution. I have already considered using the mechanisums you mention but have already seen they are somewhat limited.

I am aware of that but most of the life of a tiddler we may not need to edit the title and or text field but other fields.

  • We could reuse or extend this mechanism to allow what I am after, displaying the editable fields in the view template. We could build a set of alternate edit forms to replace the default editor but it seems to me, these end up being a monolithic edit template for each tiddler type but I am looking to get more flexibility, for example what if there were data entry form, another an update form (Each field has a button to place in edit mode independently).
  • If we follow the current mechanisms you end up with an edit form tiddler for each edit mode x the number of tiddler types. This needs more bytes and is harder to automate, or design incrementally.

This however this is possible to design, for example if the edit template searches for all fields defined in the template, or current tiddlers of the same type, it can present the editor which will create the field when a value is provided. Even existing tiddlers will receive the new field editor when a new field is added to the type.

Thanks, yes, but I am not talking about the type field, which is in effect the mime type, but the logical type of a tiddler such as contact, invoice, transaction etc… It is the primary role of that tiddler, commonly achived with tags, but I want to keep it in a standard field eg object-type.

  • The default can be used for missing fields, its created when changed to non-default, or a batch can populate tiddlers after add a new field.
  • There may be value in using a list field for “object-types” (plural) so a tiddler can be more than one type simultaneously eg contact and todo (because more information needs to be added to the contact).

It is possible to document it, but I wanted to solicit input during the design not after the event. I could document an opinionated solution when others could influence the design.

  • However this is a good approach, thanks. Do it during this project to identify the design, mechanisms then Document it, before finally coding. I will do that. Perhaps I write the first draft then get input and write a second draft for comment?

Many are there, yes, but some not yet, I will give examples because it would be hard for you to visualise. I may do some mockups.

Yes it does, thanks again for your input.

#4

Further clarification:

I am interested in building as set of features that provide a “lift yourself up by your own bootstraps” to allow immediate and incremental design where the result is always useful. This allows development to occur at the same time as populating the wiki with data you are adding.

  • This avoids the long design window before the solution is usable.
  • It allows continuous improvement.
  • The resulting databases created can be shared and reused, in whole or in part.
    • For example a field-type library can be developed and shared as a basis for other solutions.
#5

As I wrote, functionality should be discussed and not implementation details.

While brainstorming suggestions should be collected. No suggestions should be ruled out. No suggestions should be discussed. Everything is OK. Everyone is OK.

After brainstorming all suggestions need to be grouped. No discussion. No ruling out. Only grouping based on functionality.

Once all of them are grouped they can be discussed to clear communication problems. Different humans use different words for probably the same thing. Discussion yes. No ruling out.

in this phase very often it turns out that two or three suggestions actually mean the same thing. They are re-worded and the now redundant suggestions are removed.

Redundant suggestions can only be removed if the original author is ok with that. Otherwise discussion needs to go on.

Once brainstorming is done. A specification can be created.

Just some ideas.

1 Like
#6

Long existed, and many Chinese TiddlyWiki users are using these every day.

For users: SuperTag plugin, auto-generate a field editor on view template

For developers: Suggestion for all Plugin developers about field name: Let’s use Ontology to maintain Interoperability

But I think each app should have a dedicated layout, so it could works like professional full-screen app that you will see in the app store. So I’m recently creating a dedicated contacts management layout plugin.

1 Like
#7

There is an aggregation plugin that will quickly create a dynamic-table, I’ve use it for contacts management for a while.

https://keatonlao.github.io/tiddlywiki-xp/#%24%3A%2Fplugins%2Fxp%2Faggregation

But Tiddlywiki’s i18n progress is slow, so it only have Chinese version.

1 Like
#8

I think this is an excellent idea, but I’m buried in projects right now and can’t offer much help.

I want to note that this could be very complementary to my efforts in TW-SQL-Playground (discussion). That’s meant to be documentation on how to build data-centric TWs. If we also have tools to help build them, the process can only get better.

That project is back on hold for the moment, but I will return to it. In the latest version, Adding ViewTemplates was meant to next demonstrate the use of a ViewTemplate. But I floundered on the issue discussed in Using a ViewTemplate in a TestCase Widget. I will get back to that, though, as I really like the ideas, but the alternative view of the CompoundTiddler is still a little frightening for me to take on, and there’s other work to do first.

But I do really think these two efforts could end up being “two great tastes that go great together”.

1 Like
#9

I always want to create a Notion-style Database widget, based on dynamic table we already have Hyper-Table plugin, sortable, very fast, beautiful, and configuable dynamic table widgets, and enable editing of the table.

In this way, you can provide a super-tag to it, and it generate a dynamic table with predefined columns.

#10

My Kansas Railroad wiki has a kind of database structure. See Kansas Railroads (note that there are no images included because of space limitations of tiddlyhost.

I’m not a programmer nor am I a database expert so there is probably a lot done in this wiki that is not ideal. Maybe this could be used as an example either way?

Also, any comments are welcome.

1 Like
#11

Here is a DB example from TW classic:
https://baggr.tiddlyspot.com/
OR
https://twdb.tiddlyspot.com/

1 Like
#12
1 Like
#13

I would like to start this process by introducing another mode to complement view and edit. This modes working title would be update and it would start with allowing the user to select an update button just as they do edit, however the only field available initially to edit is the text field.

  • In someways this is a stream of thought reply

This update mode would be where we introduce editable forms also visible in the view template, but leaving the edit mode as is for the designer to modify the tiddler, and any field.

  • In some wikis the edit mode would not be available for users, only the update mode.
  • Is the update form a rework of the view template, with no draft tiddler or the edit mode with a draft tiddler and done button?
    • Arguable there are two possible update modes, one edit template which could use a customised edit, one based on the view template where each field provides its own view/edit mode.
  • Tagging tools, title renames and field creation/edits can be selectively be made available to the update modes, even view template.
#14

Different humans use different words for probably the same thing.

I guess that would be me. When I saw “database” in the title, I remembered TiddlyMap, which is built on vis.js, then I remembered https://fossil-scm.org which has a built-in wiki and uses sqlite as backend, then I remembered sql.js , then I wondered if this topic may mean that sonebody has the yolo idea to combine TiddlyWiki and sql.js into an awesome plugin.

#15

There is a new plugin in the making, which actually does combine TW server version with SQLite.

The MWS Plugin: Announcing the MultiWikiServer Plugin

1 Like
#16

As the author of this topic my interest in databases is specifically about leveraging the tiddler, it can act as a record in a database with the fields driving the additional columns but also comes with convenient default fields such as created and modified.

  • A simple tag or field value can indicate to which table a record belongs, and smart design can have one record appear in more than one table.
  • The advantage of such records in a database, is as written about by me in the past, tiddlywiki puts the primary key or title at eye level, so each row in a table can be considered an object within that table.
  • Tools such as relink and other design methods can easily link records, tables to build sophisticated structures, basically by providing a title (for one to one relationships), or filter (for one to many relationships), relink specifically allows us to retain referential integrity.
  • By focusing on the tiddler as a database record we automatically inherit all the features of a tiddler and can then make use of tiddlywikis UI in viewing, listing, modifying records. This includes but is not limited to plugins/shadow tiddlers as read-only tiddlers that can be edited is desired. Thus a plugin can be a repository of data tiddlers.
  • Anyone with database and software design experience can see how a number of well establish features will be needed to back this up including field definitions, field type definitions, forms, edit modes, object and report templates.

The motivation of this topic is to open access and building of common everyday databases around the tiddler as a record, and to do this providing a small subset of tools to allow such databases to be bootstrapped with minimal complexity. Provide an infrastructure to move from a tiddler, considering it a real world object (contact, task etc…) and helping automaticaly deliver the necessary tools to build and use forms as a UI into the resulting database.

  • One design effort can be generalised to help many future design efforts by capturing and publishing the core infrastructure layer to make better use of tiddlywiki as a database for everyday objects.

Futures;

Once we have a reliable and accessible way to build databases and manipulate the data therein it will become possible to build meta tools that a naive user can use to build their databases with design guidance, on how to do so effectively.

#17

As I read your replies and work offline, I have come to realise a rational approach may be to create two new view and edit modes that are adjacent to the existing view and edit template. The following are those and some working titles;

  • View template used to render the text field

    • Introduce a set of tools to present the content of a tiddler and additional fields as a view template
    • Extension to the view template;
      • Have a view template extension that allows the introduction of filed level editors within the view template form. Lets call this “view update mode”, which allows some selective updating of the details within a view template.

  • Edit template used to edit a tiddlers text field, add and edit other custom fields.

    • Introduce a set of tools to edit the tiddler that is able to take account of additional fields, their definitions and field types.
    • Extension to the Edit template;
      • Have an edit template form. Lets call this “Form edit mode”, which makes the tiddler editor able to leverage a form to change the layout of the edit mode to facilitate to better allow editing of tiddlers acting as objects and leverage forms, field definitions and field types.

  • View mode

    • View update mode
      • No draft tiddler, the field mechanism may optionally have a save and undo option or edit directly.

  • Edit mode

    • Form edit mode
      • Draft tiddler, all changes can be escaped, cancel or done to commit to store.

In the above two cases the new extended modes may be toggledon/off from within their parent mode eg View/Edit and may default under particular conditions. Thus without intervention, “View Update mode” and “Form edit mode” may default for any tiddler that has the form defined.

  • Thus toggling between these two new modes would occur as currently implemented by clicking the edit button, and return to the update mode by clicking cancel or done.
    • This toggling could be local or global
    • Within these new modes and depending on a user access configuration each of these may provide a way to toggle back to the traditional view or edit mode for a given tiddler.

It seems to me to make sense to use the existing cascade mechanism or possibly even the development of new cascades to support this.

  • For example a “view form cascade” and “edit form cascade”, the absence of which uses the current view and edit templates.
  • This will allow a cascade of forms such as simple, verbose and custom for each tiddler type or “tiddler object”.
#18

Lots of great stuff here. I wish I had more time to digest it. I want to respond to two points:

I have one tool, written for a specific project but applicable fairly generically . You can see it in action on any of the data records (in the various tables, found under All Tables in the main menu) by looking at the footer, and expanding the fields details.

(The ViewTemplate/procedure breakdown needs to be fixed. The whole summary/details structure should be part of the template to make the procedure more reusable. And that procedure should probably be split in two for the two tables shown.)

There is one point about this that I think is important. The white-list of actual data fields to be shown is carried as a field on the TagTiddler. I think that’s a useful design. It gives us a single place to capture table-level information.

I want to caution about being too optimistic here. TW is never going to replace your RDMS. It’s a tool for creating and displaying content. But that content can, in whole or in part, be relational data. And if that data doesn’t need to be used much outside this GUI, then the relational capabilities of TW are great. I love the idea of expanding them. Just so long as people aren’t thinking that this will ever replace their Oracle, MySQL, or Mongo installations.

#19

Thanks @Scott_Sauyet for you feedback, and great example.

I agree in principal, On the above item I am keen to not use tags to drive the object type or mode the tiddler is in but a kind of field cascade, an example may be a field-list that contains the list of fields and their order in the form, possibly including virtual fields for other form elements like a “ruler line”. But of course if the current tiddler does not have a field-list then go to the form or object definition to find the form definition.

  • 1.0 You could ask then what is the minimum required to define a form, is it a list of elements (Fields and form elements) that can be stored in a field, idealy this would be better than requiring a whole tiddler to define a form, why because you have to define a form tiddler.
    • 1.1 Not withstanding the above being able to set a form template is important to cover all other cases.
  • 2.0 In your example you default to a table as the form for the resulting fields, this is a useful option but is not necessarily desirable in all cases.

I agree except there a whole suit of small applications where an RDMS is not needed, yet tiddlywiki can offer most of the features of a sophisticated DB.

  • Arguably if designed carefully we could build an export method for import to an RDMS when one outgrows the tiddlywiki native DB

I think if we indicate an “object-type” for a given tiddler, then a default “form definition” will be found and used (this default could be a form similar to your table example if none has being designed. However which form definition is the default could be altered globally, eg the simple and verbose forms, or overridden locally using a “form” custom-field, or a field-list

  • This approach raises the need for a “form editor” however forms are implemented. Ideally using drag and drop reorder, which is why I like the single field form definition above #1
2 Likes
#20

@Scott_Sauyet et al, am I crazy thinking that one tiddler could have more than one object-type and example may be in a book having an object-type=chapter but also making it a todo because it needs further content or editing.

  • I think it makes sense starting with an object-type but perhaps later including object-types
  • Or perhaps todos are handled differently not so much as an object-type but as an action-type or with action-types ie any object can have an action type of todo, date etc…