Playing with Parsers

well-noted · January 3, 2025, 11:27pm

As an aside from holiday activities, and as respite from some of the other TW projects I have been working on, I have decided to begin toying around some with the audio/video parsers. While this has been on my mind for some time, it has only recently, with the implementation of MWS, become timely.

Some background: I engage with a lot of longform content, and one of the major uses of Tiddlywiki is to record, process, and identify links between notes derived from this content. It has been obvious to me for some time that having content contextually accessible within the tiddlywiki environment is necessary for a smooth workflow.

Why?

Having the ability to reference content from directly within the TW environment opens up a lot of “Shortcuts,” to the workflow, such as being able to generate notes that are explicitly timebound to content (and can trigger jumps to that location) and
I find that oftentimes, longform content can be removed or modified from wherever it is platformed, making it impossible to find the content again and requiring archival for my purposes. In this case,
I prefer and share some of the philosophical underpinnings of Tiddlywiki’s nonlinear organizational structure, making it the obvious choice for the task. Tiddlywiki’s ability to filter for and transclude content contextually is a powerful tool for the way my mind works.
oftentimes, when I am consuming this kind of content, I am listening while occupied with some other task. Stopping what I am doing to switch between multiple applications in order to take a short note becomes extremely time-and-attention prohibitive, and, as a result, I’ve ended up not taking advantage of a lot of this time when, especially, the content is actually interesting to me – defeating the point entirely.

With the MWS implementation, and the shift to databases, this has finally become, for me, a reasonable way for me to organize that kind of information within Tiddlywiki, without bogging down loading times.

So, in that vein, I have begun playing around with the core audio/video parsers to see what I could do with them

I have now set up a system whereby Tiddlywiki can replace podcast players for longform audio content

As you can see here, I have added functional timeskip buttons (standard 15s intervals)

These also work on mobile, where I’ve used the standard android media playback controls.

One way this works, for me, far better than podcasting applications, is the way that it organizes information. It’s very easy for me, as someone who consumes lots of podcasts, to lose my place in one if I switch to another, because of the way podcasts organize their information linearly.

Not a problem for tiddlywiki, so I have implemented a system whereby the audio parser updates a field of whatever tiddler it is being rendered in, to which the UI jumps whenever it is rendered:

As you can see, this timestamp is unique, but also tied to the src tiddler – therefore, the listening position survives through reloads.

Another reason for this implementation is so multiple players can be rendered in the same context and each retain its unique position:

I have implemented some of these same changes to the video parser:

These changes allow me to scrub through the video, preserving the timecode using the same type tiddler-field-specific arrangement, which is preserved and restored through reloads.

This is all very rough (though incredibly functional for me), and I have much cleanup to do just on the features already mentioned. However, this morning I have begun playing with a new audioplayer overlay for these modifications:

Ultimately, I imagine this as a draggable element that could be moved anywhere on the wiki’s screen, allowing playback of an audio or video file even if the associated tiddler is closed. While this is likely helpful in a desktop environment in some cases, I think it will especially shine for mobile uses when I am juggling multiple tasks, for example, and want to rapidly take a complex note while also switching the audio on and off and also skipping back and forth.

As I say, still playing around, much work to do, but I wanted to take a moment or two away from the project to document and share I wonder if there have been other attempts at similar TW projects from which I might take inspiration.

TW_Tones · January 4, 2025, 12:08am

@well-noted keep us informed of your progress. In a similar vein I would like to see the exif data in photographs get stored in tiddlywiki fields allowing sophisticated search, time range, even location etc…

Keep in mind before MWS the node version went a long way to acting as a scalable database, especially with external tiddlers.

well-noted · January 4, 2025, 12:48am

I self-host with node – however, one of the important keys, for me, is that I can view the same content in multiple contexts, without having multiple tiddlers, which are all tied with one another.

For example, I can open my Videoplayer Wiki now and interact with content there in a very simplified way, and my interactions will be available in all other recipes, including ones that do not actually have that bag as part of their recipe.

I have seen some methods by which people have made .tid files referential to multiple wikis, but was not convinced this wouldn’t become a massive headache 🥲 Perhaps I was being too negative tho.

Ste_W · January 4, 2025, 9:26am

Amazing stuff, makes my Christmas project of finishing the Christmas cake seem a bit trivial now!

jeremyruston · January 4, 2025, 12:28pm

Thanks for sharing @well-noted, it is fascinating to get an insight into your use of TiddlyWiki. My dream for MWS is also to be able to pour gigabytes of media into it with the ability to view that data through the lens of different TW-based applications.

arunnbabu81 · January 4, 2025, 12:40pm

@jeremyruston I hope you didn’t miss this PR by @well-noted which might be related to what he is showcasing here

well-noted · January 4, 2025, 5:02pm

This has been one of my dreams for a handful of years now, so I was very excited when I read your announcement of MWS and saw that you were basically seeking to resolve every one of the persistent challenges I’d been seeking to overcome So all kudos to you, sir.

Depends on the size of the cake

well-noted · January 4, 2025, 5:05pm

Ah, apologies, I have been dividing my time between family (read as, installing a new speaker system) and this project and have not returned to the PR – I’ll return today, I have further updates to the video parser, though I’m not sure if all them are desired as core features or if my implementation would be the preferred one (such as resuming from last-played and storing the timestamps as a field value).

If there is a sufficient desire for them to be core features, I’ll continue working that angle, otherwise I may package as a plugin.

@arunnbabu81 in regard to you annotation tool, I will be very interested in taking more of a look at it when I have a chance.

My idea is to set up a custom streams node for tiddlers tagged as media and have new nodes adopt the timestamp value from their parent: in that case, they could each be transcluded through a viewtemplate that includes a button which changes the value of the parent tiddler to match, and then clicking the button would jump to that time.

The only conceptual problem I have with that is, because the timestamp fields are src-specific, I am unsure how I’m going to write a code that would say “Take whatever field from the parent which has the prefix timestamp” – I’m sure it’s possible, I’ve just never set a field value according to the field prefix of another tiddler.

well-noted · January 4, 2025, 5:33pm

@buggyj, in response to your question on github

vivaldi_5dRJIm1rID-ezgif.com-resize

Changes I’ve made to the parser allow an embedded or canonical video to buffer and jump to multiple timestamps.

You can see in this example that there is a brief loading period whenever the time is paused or the time changes – that is because the element is reloading each time the timestamp field of the tiddler changes: however, since working on that, I have figured out how to bypass the reloading while working on the audioparser, and I believe I should be able to use the same strategy for the videoparser.

etardiff · January 4, 2025, 6:20pm

I’d love to see your work on this (either as a plugin or as a core feature, TBH). I’d hacked together a simple interface that lets me skip to audio timestamps, but not solved the reloading issue. Everything you’ve mentioned here sounds like a great QOL improvement!

well-noted · January 5, 2025, 1:36am

That was a pretty persistent problem for me too, even after I’d taken steps to prevent it – however, on a hunch, I realized I could overcome it simply by transcluding the video: it reloads each time the src tiddler is modified, so when the timestamp field of the src updates, there’s no way around it – in retrospect, I realize that’s because the way tiddlers are “modified” is that they are actually deleted and then replaced by a tiddler with the original content+modifications, so each time it seemed that the element was reloading, it was actually just losing access to the src for that short period of time.

I’ve got the video and audio parsers both working, I’ll post updates here and to github tomorrow

I’d be interested in seeing your hack.

etardiff · January 5, 2025, 2:22am

It’s not particularly sophisticated — partly because I wasn’t sure how to jump to timestamps other than by modifying the <audio src=>. My template is designed to display one audio file per tiddler, with a link/local file path stored in the audio field. The text field is reserved for timestamped notes (so they’ll show up in standard searches).

In “view” mode, clicking a (green) timestamp takes you to the appropriate point (but reloads the audio element in the process). There’s no current mechanism for jumping back/ahead or saving the current timestamp; I hadn’t figured out how to retrieve that information.

Inline edit mode (modifying the text field of the current tiddler):

The template contains a few global functions/macros (defined elsewhere), but I think it’s probably self-explanatory enough:

\function timestamp() $:/state/timestamp/ [<currentTiddler>] +[join[]]
\function time() [<timestamp>get[text]] ~0 
\function src() [{!!audio}] #t= [<time>] +[join[]]

\function get.seg(n) [<start>split[:]nth<n>trim:prefix[0]]
\function jump.to() [get.seg[1]multiply[3600]] [get.seg[2]multiply[60]] [get.seg[3]] +[sum[]]

\define jump() <$action-setfield $tiddler=<<timestamp>> text=<<jump.to>> />

\define edit()
@@.tc-tiny-gap-left
<% if [{!!editing}!match[]] %>
	<<done-button>>
<% else %>
	<<edit-button>>
<% endif %>
@@
\end

!!! Timestamps <<edit>>
<div class="timestamps-list">
<% if [{!!editing}!match[]] %>
	<<edit-textarea text
			btnClass:"hide"
			placeholder:"""00:05:08 - Topic
00:15:20 - Another topic
01:32:54 - Yet another topic
""">>
<% else %>
<$list filter="[split.line{!!text}!match[]trim[ ]]" variable="line">
<$let
	start={{{ [<line>split[ - ]first[]] }}}
	label={{{ [<line>split[ - ]butfirst[]join[ - ]] }}}
>
	@@.txt-r
		<$button actions=<<jump>> class="tc-btn-invisible tc-tiddlylink">
			<<start>>
		</$button>
	@@
	@@.e <<label>>@@
</$let>
</$list>
<% endif %>
</div>
<div class="audio-player">
<audio controls
	preload=metadata
	src=<<src>>
/>
</div>

arunnbabu81 · January 5, 2025, 5:36pm

@well-noted two things in which I am interested are

To know the current playtime of each video (html videos embedded in separate tiddlers) so that I can do annotation easily.
A way to fix the video refresh issues

well-noted · January 5, 2025, 6:09pm

New version of the video parser committed to PR, you can see in this preview that it significantly improves upon the version previewed yesterday:

vivaldi_J7wOsRNP5b-ezgif.com-resize

Additional note, one will need to transclude the src tiddler into another tiddler for this to work, because if one is viewing the video within the src, then, when the timestamp updates, the src is temporarily interrupted – so if this were to merged, therefore, we would need to make additional changes so that, if the current tiddler is the src tiddler, it would not store timestamps. Personally, this is a small compromise for me, so I think it reasonable to use right away, in which case you would want to replace the core videoparser with this latest version of the “improved parser”

well-noted · January 5, 2025, 6:15pm

Nothing I’ve done will have any effect on iframe videos – I had thought to commit some time to that, since iframes have been traditionally how I’ve handled this, however, as I’ve briefly noted earlier, I’ve become distrustful of video hosts (Youtube) to preserve unadulterated content from the original – therefore I archive what I want to reference, and knowing MWS was on the way, was waiting to do something along these lines.

If I were to attempt something with youtube links, I would probably attempt to retrieve and store the timestamp as reference values within individual note tiddlers, and then have some button that would modify a variable within the construction of the iframe (which might get very complicated, it certainly sounds like a headache, though I admit it may be my personal biases).

arunnbabu81 · January 5, 2025, 6:29pm

I was not talking about iframe videos, but html video tag

<video controls src="" type="video/mp4">

I am using a viewtemplate to display the html video tag with src value stored in a tiddler field

well-noted · January 5, 2025, 7:19pm

@etardiff, on some reflection, I think the audioparser will need to be a plugin because it includes code that is necessary to make the audiocontrols work too – unless there were some interest in merging the audiocontrols overlay into the core

Was going to make a pretty package, but then realized I will need to make some file changes for that, so in the meantime, you’ll need

You’ll also this CSS stylesheet if you decide to use the audiocontrols
ImprovedAudioParser.css.tid (3.3 KB)

–I believe that the parser will work without the audiocontrols (just don’t include that file in the parsers folder) but there might be dependencies. If you decide to try, let me know – I could probably do a refactoring and/or add the ability to toggle the control overlay.

Per usual, lmk if you encounter any other problems and I’ll try to address in a timely manner

well-noted · January 5, 2025, 7:49pm

Lmk if the recent commit gives you the tools you need, and what additional features would be helpful. I haven’t begun working on the annotation part of the tool yet, but it is high on my priorities list, so hearing what your requirements are now might be foundational to me thinking.

arunnbabu81 · January 6, 2025, 2:47am

This is a demo I had previously shared - it is a wikitext based basic annotation template (it was based on a similar work by telumire and buggyj). What was missing in that concept was the ability to add start and end time of a annotation segment easily since the current playtime value is not stored anywhere. Refresh issues was the second problem i encountered. I didn’t get time to test your updated video parser. I will check it once I am back home. It would have been easy, if there was a demo wiki.

etardiff · January 6, 2025, 4:54am

Interesting, thanks for sharing! I don’t think it will fit into my workflow (since I haven’t been making audio-type tiddlers), but I tested it on TW-com and the controls look quite slick.

I’m not sure how I feel conceptually about the timestamps being stored as fields on the host tiddler itself. TW tends to use $:/state tiddlers to store UI interactions (like unfolding a section of a TOC macro, for instance), and this feels like a comparable situation.

IMO, playing an audio/video file shouldn’t “dirty” the parent tiddler; I should be able to import it into a different wiki without saving (potentially, someone else’s) timestamps.
I don’t love the qualified field names, either; they seem likely to result in a lot of field pollution. Using state tiddlers would let you keep the timestamp in the text field(s) and avoid this issue.

The floating player is a nice touch! I do find myself wishing that I could scrub through it as you can with the embedded players, but I don’t know whether that’s feasible, since you’re already using click-and-drag to reposition it in the parent window.