Best approach to restructuring (adding dimension) to a project?

#1

Here’s a familiar kind of growing pain:

Suppose I’ve been working with a “flat” TiddlyWiki design — a glorified spreadsheet:

  • 60 tiddlers, each representing a student
  • 6 fields for each tiddler, in which to track student’s assignment grades.

Then an awkward moment happens: After entering and working with data for a while in this “flat” model, I realize each of those data “points” (field values) deserves a tiddler. (Say, I realize I want to record qualitative comments, submission date, and even the content of what was submitted for each assignment.) So, I now want a project with 360 “atomic” tiddlers (plus the 60 per-student tiddlers, which will still serve as summary nodes). And… I want not to lose my data.

Surely there’s a word for this kind of “growing pain” in data handling. It’s adding a dimension, like “exploding” a 2-D graph into a 3-D graph – exactly what distinguishes a “flat file” database from a relational database.

If I had an infinitely patient employee (or perhaps if I knew how to recruit ChatGPT to work effectively with json data), I’d just say:

For each assignment field of each student tiddler, create one new “submission” tiddler. Come back to me with 360 tiddlers that look like this:

title: OldTiddlerTitle_FieldName
text: OldFieldValue

Have you encountered this hurdle, mid-project? Is there anything like a general solution for this task?

-Springer

#2

When I need to do extensive refactoring, I nearly always end up building myself a temporary tiddler with a list widget and a custom button. In this case, I’d probably start out like this:

<$list filter="[tag[student]] +[limit[5]]" variable="student"> <!-- limit so you don't have to render 60 items at once -->

<$link to=<<student>> />
<$button>Separate
<$list filter="a b c d e f" variable="field"> <!-- where a, b, c, d, e, f are the assignment fields-->
<$action-createtiddler
    $basetitle={{{ [<student>] [<field>] +[join[_]] }}}
    text={{{ [<student>get<field>] }}}
    tags="submission"
/>
</$list>
</$button>

<$list filter="[tag[submission]prefix<student>]"> <!-- this list is just to check our results when we press the button; remove it once you're confident it's working! -->

* <$link />
</$list>
<br>
</$list>

This will give you one button per “starting” tiddler, which is usually how I like to start: it makes it easier to catch any errors. In this case, if you don’t want to retain the student tiddlers after you’ve generated the six corresponding assignment tiddlers, I’d probably also add <$action-deletetiddler $tiddler=<<student>> /> between </$list> and </$button>—after you’ve verified that you’re getting the results you want, of course. That will remove “already-processed” students from the queue, so you won’t have to modify the limit in the outer list filter.

If you’re not converting too many tiddlers (or you don’t mind clicking 60 buttons), you could stop here! Alternately, if you’re feeling brave and recently-backed-up, you can remove the <$link/>s and <br> and move your <$button> tags outside the outer list:

<$button>Separate
<$list filter="[tag[student]]" variable="student">
<$list filter="a b c d e f" variable="field">
<$action-createtiddler
    $basetitle={{{ [<student>] [<field>] +[join[_]] }}}
    text={{{ [<student>get<field>] }}}
    tags="submission"
/>
</$list>
</$list>
</$button>

You can just delete the button tiddler when you’re done with it, or save it to modify next time you need to do some batch-processing.

5 Likes
#3

Super, this got me started!

I decided to process by fields. Partly because my actual case is more complex, and I wanted to build a net to “catch” all the custom fields in the project, rather than typing out a list.

Also, in my experience it’s not uncommon for just a subset of fields to need a “break-out” solution (and to require it all across the solution), while it’s harder to imagine only some records needing all their fields broken out into separate tiddlers (though perhaps in the case of a project with some merely “archive” tiddlers, one would not bother adding in the details that active records require?).

Here’s what I came up with (framed in a way that I hope is maximally neutral for adaptation to others’ purposes):

<$list filter="[all[]fields[]sort[title]]-[all[shadows]removeprefix[$:/language/Docs/Fields/]]" variable="field">

<$button>Separate ''<<field>>'' field: Create '' <$text text={{{ [has<field>count[]] }}}/> '' tiddlers:<br>       title: original_title_<<field>> <br>     text: original field-value
<$list filter="[has<field>]" variable="record"> 
<$action-createtiddler
    $basetitle={{{ [<record>] [<field>] +[join[_]] }}}
    text={{{ [<record>get<field>] }}}
    tags="detail"
/>
</$list>
</$button> 
(<$text text={{{ [suffix<field>count[]]  }}}/> tiddlers now have ''<<field>>'' in title) 
<$button> Delete ''<<field>>'' field 
<$list filter="[has<field>]" variable="record"> 
<$action-deletefield $tiddler=<<record>> $field=<<field>>/> </$list>
</$button>
</$list>```
2 Likes
#4

Not sure, but I wonder if Tiddler Commander can do that… @Mohammad ?

#5

I think the approach by @etardiff is better here! It is flexible. But Tiddler Commander has two nice features here can help

  • Add/remove fields to bulk of tiddlers or replace field values
  • Inspect a filtered list of tiddlers and view/edit their values in one place
1 Like
#6

Lessons Learned from @Springer case :wink:

  • Before start entering huge amount of data, decide and design a good data structure
  • Enter few amount of data and test your design
2 Likes
#7

For me that’s not how real-world systems grow. Sure, you do up-front design and try to fit into it, but priorities change all the time, and sometimes changing priorities cause you need to redesign some of it. I didn’t answer the original question because my own technique for this is too specific to my workflow – using Node to bulk-modify tiddlers. I’m guessing that @springer would prefer an in-wiki technique, as @etardiff demonstrated. But if you want, I can demonstrate how I would do it with Node.

2 Likes
#8

Of course careful planning and designing in advance is ideal… for exactly those cases and details that turn out (from the current point of view) to be important.

Though I hope this thread is useful to others, I’m not happy to summarize the conversation as a “lesson learned” :thinking: — at least not in the way that “lesson learned” can imply a resolve to “do better next time”. (I do see, @Mohammad, that your wink offsets the lesson-teaching words!)

Some projects (especially professional ones) begin with a clear big picture, while other projects develop over time. In many concrete cases, the workable first step is the tentative and sketchy first step…

Some people by temperament are judicious planners (Myers-Briggs Type J), who decide what to do, and do those things deliberately and carefully. Others are “Prospectors” — people who tend to begin more projects than they finish, adjusting along the way which ones to invest more in, thriving through improvising and tinkering.

One of the joys of TiddlyWiki, for folks with this latter temperament (but also for judicious planners who discover unexpected complexity) is its flexibility and expandability.

That’s always my first tool of choice! Most design-fixes can indeed happen “on the fly” with the help of @Mohammad’s Commander plugin.

I was motivated to post, yesterday, because this dimensional expansion task is just about the only design-adjustment challenge for which Commander is not already — out of the box — the best tool for the job. :rocket: I’m tremendously grateful for it! :pray:

-Springer

2 Likes
#9

If you wish to pursue the “more dimensions” approach, you might want to consider a simple view of adding dimension(s)

Start with a Matrix (a 2-D array) that consists of 3 rows of 4 columns.

1 2 3 4
5 6 7 8
9 10 11 12

(This renders very well if built as a simple wiki table using the | markup)

The add another “sheet” (spread sheet term) to create a 2 X 3 X 4 (3D array) for

01 02 03 04
05 06 07 08
09 10 11 12

13 14 15 16
17 18 19 20
21 22 23 24

This can be extended to much larger
(Unicode does 128x256x256x256 cells that are accessed using coordinates for each dimension of Group,Sheet,Row,Column)

Any singe cell can be read by using its calculated coordinates
Cells in a dimension (vectors) can be accessed via enlist.
Nested enlists can be used to iterate higher structure levels.

This is theoretically completely extensible, so it make a good “design”, especially in TiddlyWiki’s XML environments since there is no need to allocate memory space for any empty cells.

Personally, watching Jeremy’s developments for some time has convinced me that I may yet get an effective re-implementation of Ken Iverson’s APL.

#10

I do occasionally, but try to avoid this, however it is sometimes inevitable my key refactoring tools are;

Novice users do not have the skills, for bespoke refactoring, and the community can help, but unless they share the wiki, or a minimal case (which they are not very good at - yet) then it gets quite messy and tends to demand a lot of volunteer time.

  • Also refactor to what?, it sometimes needs a lot of skill to design complex relationships, so they are not broken in some way,

Not withstanding these issues, it is possibly easier to refactor in tiddlywiki than any other code or platform.

#11

Following your link, I saw that I had read and commented on that thread a year ago, when you first posted it. Funny how much difference a metaphorical term (like “refactoring” — which had not occurred to me) can make to whether a search turns up something.

#12

I do see how numerical placeholders elucidate the structure of an N-dimensional array… in this case, I’m working with a natural-language wiki (without much structure beyond intuitive titles and field names). For a complex specialized project — where unique identifiers could be set up in numerical arrays and connected by mathematical filter operators — it would be fascinating to consider how dimensional expansion could be more extensively developed.

#13

For me, its a “slippery slope” I ventured onto in the 1970’s.

TiddlyWiki revitalized my interest in “Information Density” (squeezing as much information into the shortest Strings possible) by providing excellent Tag support that spans most of the UniCode space. Of course, since Tags are also Variable Names and Values that can have precise meanings, this leads to building a personal Dictionary of single character word “meanings” using an alphabet of millions of characters.

I know this sounds esoteric, but TiddlyWiki’s transclusions and Data Dictionaries make it all wondrously efficient.

I am not “selling” the use of designs based on this, but I can confidently state that it is a general solution, with mathematical rigor, that will likely exceed all expectations.