JSON string on clipboard: custom deserializer for webhook result

#1

Here I’m venturing past my skillset again, but someone here will tell me what’s possible.

I’m working with lots of handwritten documents, and handwritingocr.com seems to have decently accurate transcription, and it can be set to interact with webhooks to output JSON strings.

I still need to learn more about webhooks, but I was able to process one handwritten page-image and have the transcription results appear at my bespoke webhook URL as a JSON string. Cool!

I helped myself to the “paste JSON from clipboard” solution from this thread, and …

… needed to go do some edits in BBEdit to add outer square brackets, to assign “title” instead of “file_name” … remove some nested structure I don’t need… and rename “transcript” to “text”…

OK! Proof of concept worked: I got the transcript with various bits of metadata imported as a tiddler.

AND… I did climb a learning curve for custom deserializers before (coached along by saq back when I started a workflow that pulled things in from a moodle export, and again with bibtex import process). So I think I understand how I can clone a deserializer and convince it to treat the file_name key as the title field, and to parse the date fields which aren’t in tiddlywiki format…

BUT this JSON on my clipboard has a nested structure that buries what I’d like to see in text field within an additional layer. Does anyone know whether/how to tweak a deserializer so that it can “dig” into the nested key (called results), ignore its page_number value, and either just take the first “transcript” value, or (ideally) concatenate the value of all the transcript keys found within… ?

Is this the kind of thing that’s possible, and that folks can coach me through?

(I’ll save other webhook-related questions for another day.)

Many thanks in advance!

#2

Hi @Springer, Funny, I am working on the same stuff. I wrote a moodle-plugin to allow students posting their handwritten work.

#3

I am not 100% sure, what you try to achieve.

Do you want to convert hand written documents into standard tiddlers, or do you need specialised formats. You did mention moodle and bibtex

I think everything you describe here can be done using a CLI LLM like Gemini or any other app, that you prefer.

In 2022, I did download a PDF which contains ~20 hand-written pages. Every page also includes drawings. Some text was really hard for me to decipher. … So I did spend hours to read and create a “streams-like” outline, with the basic summary points in it. The details left open for later.

Later 2025 - I did tell Gemini (free version) to analyse the content and create Markdown files, because most LLMs can handle them out of the box.

I did split the big PDF in to several PDFs. 1 PDF for 1 page with a certain naming. I told the LLM to read the PDFs analyse them and create a .md file with the same name as the PDF.

I did test my instructions with 1 file first. When I was happy with the result I told the LLM to read all files …

It needed about 15 minutes (and used all the free tokens for the day ;), to convert the PDFs into text and also add “context” descriptions to the output files. The OCR results are perfect. It even managed to read stuff, that I could not read at all.

So the .md files are perfectly usable.

It would be possible to let the LLM convert them into WikiText, but including the markdown-plugin is easier.

What I want to say. I think current LLMs are perfectly fine to be used for the usecase you describe. There would be no need to use the clipboard, with a JSON format. It should be possible to create .tid files directly.

With the right prompt it could get your hand-written texts, convert it into .md or .tid files, that could directly be imported into a TW.

If you need bibtex fiels, that should be possible too.

Just some thoughts.

1 Like
#4

Very cool!

Unfortunately my IT admins are very nervous about moodle plugins, and made me jump through many hoops when I wanted to install Generico, and even when I wanted to make a tiny tweak to the css of the university’s default (Boost) theme.

Still, please do keep me posted about your experience.

I’m of two minds about how I handle and process handwritten work. (I’m not of two minds about asking for handwritten submissions on short assignments, though. On that much I’m clear.)

On one hand, I’d like to offer hand-written comments in response, to reinforce the substantive points I’m making about the value of slowing down to read from paper copies and to respond with pen and paper (and to learn to write with a patience and rhythm that make it legible!). There’s lots of depth to generating and interpreting analog penstrokes, being more present to the embodied and muscular process of writing as it unfolds in time and across the page, etc.

On the other hand, having all the data-handling affordances of a digitized workflow (and copy-pasting the same feedback text to the three students who need the same critical nudge) is a big convenience to relinquish!

#5

I wanted to get eScriptorium running on our server to do the recognition: https://sofer.info/

1 Like