Json from moodle, 2 questions/requests about non-ideal json files

#1

Hello all! Turns out moodle (university course-materials platform) exports json. Hooray! (Probably other courseware can do the same.) What I know about json, however, is limited to what I’ve learned through TW tinkering.

Small nuisance: the json that moodle exports needs to be massaged in 2 ways:

(1) it has no title field, so I need a find-and-replace on the json to turn “message:” into “title:”
(2) it has lots of numeric fields that lack the quote marks for text strings. And in my experience so far, the TW import process just ignores fields that fail to include quote-marks around the value. So I need to fire up some editing tool to go in there and do a regular-expression search-and-replace, or something like that, to ensure that numeric field values get quote marks around them.

It’s not a big deal in theory. But I envision a workflow where I’m grabbing these moodle-exported json files and dragging them into my tiddlywiki pretty regularly.

So, my questions are:

(a) Is there any chance the import process could recognize that an incoming JSON lacks title fields for its records, and could then offer to pull from a different specified field (like “subject” or “message” or “name”, as user specifies) to populate TW’s title field?

(b) Is there any reason tiddlywiki couldn’t be tweaked to accept unquoted numeric values as if they arrived within quote marks?

(c) If the above tweaks are unrealistic: Is there any simple tool y’all know of that can transform something like a json document, given particular find-and-replace rules, so that I don’t have to open the json and peck around each time? (I’m working on a mac desktop, and can get 80% of what I need with TCleaner Pro, but it still requires opening the file and a couple commands each time.)

[For completeness’ sake: moodle’s json output also has an extra bracket at beginning and end [[{ ... }]] that seems to break the import process. Easy as heck to remove, but tedious when there’s many of them.]

Many thanks!

-Springer

#2

We can definitely try to create a custom deserializer (to interpret the JSON) for Moodle imports given enough information on how to map the fields.

Is there a fixed field that should always be used as the title? Or a series of fields attempted one at a time until there is match? Or do you need to be able to specify the field when importing?

If you aren’t in a particular rush and can provide some example files and information, I can find time to take a look towards the end of the year.

#3

@Springer consider looking at JSON Mangler for importing non-tiddlywiki tiddler JSON and other files.

As @saqimtiaz said, We may also be able to give you alternative answers and methods if you share a sample of a JSON you are wanting to convert.

#4

Saq and Tones,

Thanks to you both for friendly replies!

My impression about JSON Mangler was that it was intended mostly for handling tiddlers with JSON type (as opposed to unbundling the JSON upon import). But perhaps I need to learn more about it.

I too am plenty busy until the end of the academic year, and can certainly muddle through ok with firing up a text-editor for search-and-replace, for now… I’ll return to post a sample moodle-exported json when I can.

-Springer

#5

Actualy, i would say tiddlywiki is built to use JSON for storing tiddlers. To handle other JSON files comprehencivly you need to write your own or use other tools such as JSON Mangler.

@joshuafontany ?

#6

Yes just get in touch when you have the opportunity. I think we can come up with a custom deserializer, which would basically mean you can drag and drop in a custom dropzone to import without extra steps needed.

#7

Tones and Saq (and anyone else following the moodle-json-export to tiddlywiki-import challenge),

Here’s a sample 2-record json exported from moodle, showing the many unquoted numeric fields, and the extra brackets at beginning and end. The subject field is the one that I would normally import as title… BUT perhaps it’s wiser to use the id field and map “subject” onto “caption” to handle potential case of duplicates, though I do love natural-language titles… Replies in a forum do fill the parent field with the id of the original post. (The drag about using the id number as tiddler title is that I sometimes generate records through my own workflow, and making up an arbitrary unique id number is one extra step. Meh.)

Not all the numeric values matter much, but having them all come in as quoted text strings would be helpful (I may be surprised by what I can do with the metadata later).

It would be lovely to get the created and modified dates to translate, but that’s surely more trouble than it’s worth. (But just for completeness’ sake: the created/modified dates can also be exported in epoch format, so that’s 1637445030 for the first record below – but again, without quotes, so tiddlywiki won’t pull it in unless there’s a special parser or regexp conversion.)

[[{
    "id": 541563,
    "discussion": 289911,
    "parent": 0,
    "userid": 24513,
    "userfullname": "JF",
    "created": "Saturday, November 20, 2021, 4:50 PM",
    "modified": "Saturday, November 20, 2021, 4:50 PM",
    "mailed": 1,
    "subject": "JF's Essay",
    "message": "Here's where the content of the submission sits, which usually comes in with plenty of html and special characters. Moodle seems to handle the json export of html message body content successfully in my experience so far (using proper escape character sequences). Of course the wordcount and charcount below are incorrect because the text here is pasted in...",
    "messageformat": 1,
    "messagetrust": 0,
    "attachment": 0,
    "totalscore": 0,
    "mailnow": 0,
    "deleted": 0,
    "privatereplyto": 0,
    "privatereplytofullname": null,
    "wordcount": 248,
    "charcount": 1255
},{
    "id": 542779,
    "discussion": 290651,
    "parent": 0,
    "userid": 18237,
    "userfullname": "CR",
    "created": "Tuesday, November 23, 2021, 1:12 AM",
    "modified": "Tuesday, November 23, 2021, 1:12 AM",
    "mailed": 1,
    "subject": "Kant, Nietzsche, Beauvoir",
    "message": "Here's another submission's body text here...",
    "messageformat": 1,
    "messagetrust": 0,
    "attachment": 0,
    "totalscore": 0,
    "mailnow": 0,
    "deleted": 0,
    "privatereplyto": 0,
    "privatereplytofullname": null,
    "wordcount": 185,
    "charcount": 1004
}]]
#8

What would you want to set the type field to? Default? Markdown?

#9

Also, do we know what timezone the created and modified dates are in?

#10

Saq, Default type is fine; times are GMT-5.

#11

OK I think I have something close to usable. Github however is currently down so I cannot share until they resolve their issues. Will upload in the morning.

As is often the case, the bulk of the time spent on this went into chasing what might be a core bug in timezone handling in the JavaScript method$tw.utils.formatDateString.

#12

Let me know if this seems like a good workflow, and if so what is missing. There may well be bugs, minimal testing performed:

https://moodle-import.tiddlyhost.com/

1 Like
Creating a tiddler from a JSON *string*
#13

Saq, this import flow works well, at least on quick testing. Thanks so much!

The dates come through well, so long as I use the “human-readable” version on export. I forgot to ask that userfullname be mapped onto modifier field, but I can tweak that as needed.

I do think it’s likely that others may encounter .json files with unquoted numeric values, and/or json files with epoch-format dates…

So, for those thinking about json imports in the long term, perhaps allowing such options (as well as designating fields to map onto standard ones like title, modifier, modified, text) would be helpful for future users.

But I’m very happy and grateful!!

-Springer

#14

@Springer working on this also put me in mind of a configurable JSON import where the user can specify the field mapping.

However, for that to be truly useful the core import mechanism needs to be rewritten so that once the import tiddler has been created, it is possible to change how the data being imported is parsed without having to delete and re-import. So imagine that with the import dialog in front of you, you can change the settings for how to parse/import the data and you can preview how that changes the tiddlers to be imported in real time.

When I get the chance I will add the userfullname mapping as well as an option for the input date format.

#15

@saqimtiaz there is a sophisticated input form process in JSON Mangler already. I have used it a lot for CSV but I understood it translates JSON as well (not proven).

Just in case it helps.