Suggestions for visualisation of big field experiment dataset with tiddlywiki

#1

I am working on a project to analyse very big field experiments in agriculture (thousands of trials, tens of thousands of plots) and plan to visualise them with all related information.

The methods I used before (e.g. C# asp core, R Shiny) depends on setting a web server. I give up this method now as the overhead to maintain a server in long term.

Now I think TW would be the best tool for my project for data visualisation. I could use R to generate the all static pages (tens of thousands). However, the generated single HTML file will be very huge.

My idea would be to store all data as JSON tiddlers or fields, then access through view template to visualise them as card, tables, and figures.

  • Should I use JSON tiddlers or tiddler fields?
  • Are any requirements about JSON data structures in TW for easier access?
  • Should I only use wikitext or mix with javascript?

Thanks for any suggestions.

Replace text using field value in a macro
#2

Hi, I am a new TW explorer. maybe i can help

Tiddlers json just need “title” field, then “text”, “tag” are show and use by UI.
all other fields are yours

Not sure about the question

What would be the average size of your trials data, what kind of data structure?
Maybe you could consider using “ipfs” to reduce TW size. This way TW can be overloaded Tiddlywiki and the IPFS “Video Wall” Experiment

#3

My general approach would be to use a seperate wiki “Data import wiki” and the JSON mangler plugin to import the data from CSV or JSON and convert into tiddlers within a plugin. Then transfer these data plugins to my “visualisation wiki”.

  • I would do this first and investigate the resulting data size 10’s of megabytes will be fine. Only include data you will use. I can help if you share such data privately (non disclosure) to evaluate its practical use.

In the visualisation wiki I would make use of standard and advanced wiki text to list, filter, review, navigate, report, display, analyse the data.

  • Almost no need for any custom javascript
  • Plugin - Update / Replace data easily inc’ a reduce set while designing you wiki

Performance and size issues are far less of a concern than you think, with many ways to optimise and improve performance even with large data sets in single file wikis.

  • If you want to operate on data, even in a server you will need the data in memory.
  • Users dependant on TiddlyWiki running in a browser should consider giving their browser permissions/setting to make use of more memory, most browsers restrict the resources they use, so as not to impact other APPS, however big TiddlyWiki users app tend to be in the browser.
  • The data will exist in shadow tiddler within the data plugin on which you can use native tiddlywiki to handle.
  • Once you have your visualisations etc… you could publish an interactive wiki or static representations.

Additional notes;

#4

@papiche and @TW_Tones Thanks for your suggestions.

Data plugin is a nice idea to separate data with normal wiki. I would not worry too much about data importing part. I use R package rtiddlywiki to convert all required data into tiddler format.

My data is not very big (< 100 MB). I believe the browser can handle it as my colleague is developing another visualisation web app to handle GB data in memory.

The challenge is my data is unstructured and comes from several sources.

My main concern is too many tiddlers as I have thousands of experiments, more than 10 K tiddlers if I generate all visualisation in separate tiddlers.

#5

You said so. So you’ll have to figure out some tags and field values that could be defining your samples. Then “post treatment” and data processing will reveal your sauce taste :wink:

#6

This could be a challenge but if there is sufficient information in your data you can just use search and write queries to find what you want. You could just import the various sources as plain text files, ie totally unstructured.

  • The key tip is import them in a way you don’t loose any information, like keys, links, reference info, internal structure like row or column breaks, string delimiters etc…
  • TiddlyWiki’s ability to store plain text tiddlers may actually be a strength here that has not yet being tapped into, because you just get the data in (unstructured) and then start to use it, if no other way than through smart search filters.
  • With out using the tiddlywiki built in parsers, I have written a range of csv or text parsers that make use of [{input-tiddler}splitregexp[\n]] and similar to extract lines from text then process each line. Ultimatly into tiddlers or strings or calculations. MAP and REDUCE operators are also important here.
  • If you thus restructure your input to more structured forms, use macros such that you can add or update new data as its added/updated.
    • Always consider more than one tiddlywiki, such as one for import, convert and structure and the other for working with the data. A “Data Ingestion wiki”
  • Many of tiddlywiki’s features actually favour what you are trying to do, and many and I have said before “tiddlywiki is a great way to explore, research and discover relationships in your data”.

I suggest don’t worry about this until you get performance issues and then ask for help. I worked on a 66,000 tiddler wiki we got working with optimisation, but in this case it was for a dictionary so each tiddler was small.

  • If you really can’t handle the additional overhead of seperate tiddlers (not much more than created and modified fields) use a JSON tiddler for each data set with common rows. Again this can be done in your “Data Ingestion wiki”.

I can be engaged professional on this matter but will always do my best to help the community for free (within reason)

#7

@TW_Tones Very appreciate it for your suggestions. I start to work on the prototype and privately talk to you later if I need any professional advices.

1 Like
#8

Hi Zheng, Would you mind sharing your final results when it’s done? Don’t need to show your real data, just the way you did it. Handling big data is always a fascinating topic and if there is a chance to make TW do it easily, it would be fascinating. One more filed where TW could be used that I never thought and that can unfold in a lot of opportunities, not just for the community but for @jeremyruston too, who knows! (just wandering here!)

2 Likes
#9

Sure will do if I come out something.