Playing with Parsers

[well-noted]

@TW_Tones

Keep in mind before MWS the node version went a long way to acting as a scalable database, especially with external tiddlers.

I self-host with node – however, one of the important keys, for me, is that I can view the same content in multiple contexts, without having multiple tiddlers, which are all tied with one another.

For example, I can open my Videoplayer Wiki now and interact with content there in a very simplified way, and my interactions will be available in all other recipes, including ones that do not actually have that bag as part of their recipe.

I have seen some methods by which people have made .tid files referential to multiple wikis, but was not convinced this wouldn’t become a massive headache 🥲 Perhaps I was being too negative tho.

[Ste_W]

Amazing stuff, makes my Christmas project of finishing the Christmas cake seem a bit trivial now!

[jeremyruston]

Thanks for sharing @well-noted, it is fascinating to get an insight into your use of TiddlyWiki. My dream for MWS is also to be able to pour gigabytes of media into it with the ability to view that data through the lens of different TW-based applications.

[arunnbabu81]

Allow seeking throughout video files

@jeremyruston I hope you didn’t miss this PR by @well-noted which might be related to what he is showcasing here

[well-noted]

@jeremyruston

My dream for MWS is also to be able to pour gigabytes of media into it with the ability to view that data through the lens of different TW-based applications

This has been one of my dreams for a handful of years now, so I was very excited when I read your announcement of MWS and saw that you were basically seeking to resolve every one of the persistent challenges I’d been seeking to overcome So all kudos to you, sir.

@Ste_W

makes my Christmas project of finishing the Christmas cake seem a bit trivial

Depends on the size of the cake

[well-noted]

Ah, apologies, I have been dividing my time between family (read as, installing a new speaker system) and this project and have not returned to the PR – I’ll return today, I have further updates to the video parser, though I’m not sure if all them are desired as core features or if my implementation would be the preferred one (such as resuming from last-played and storing the timestamps as a field value).

If there is a sufficient desire for them to be core features, I’ll continue working that angle, otherwise I may package as a plugin.

@arunnbabu81 in regard to you annotation tool, I will be very interested in taking more of a look at it when I have a chance.

My idea is to set up a custom streams node for tiddlers tagged as media and have new nodes adopt the timestamp value from their parent: in that case, they could each be transcluded through a viewtemplate that includes a button which changes the value of the parent tiddler to match, and then clicking the button would jump to that time.

The only conceptual problem I have with that is, because the timestamp fields are src-specific, I am unsure how I’m going to write a code that would say “Take whatever field from the parent which has the prefix timestamp” – I’m sure it’s possible, I’ve just never set a field value according to the field prefix of another tiddler.

[well-noted]

@buggyj, in response to your question on github

Changes I’ve made to the parser allow an embedded or canonical video to buffer and jump to multiple timestamps.

You can see in this example that there is a brief loading period whenever the time is paused or the time changes – that is because the element is reloading each time the timestamp field of the tiddler changes: however, since working on that, I have figured out how to bypass the reloading while working on the audioparser, and I believe I should be able to use the same strategy for the videoparser.

[etardiff]

@well-noted

I have figured out how to bypass the reloading while working on the audioparser, and I believe I should be able to use the same strategy for the videoparser.

I’d love to see your work on this (either as a plugin or as a core feature, TBH). I’d hacked together a simple interface that lets me skip to audio timestamps, but not solved the reloading issue. Everything you’ve mentioned here sounds like a great QOL improvement!

[well-noted]

@etardiff

I’d hacked together a simple interface that lets me skip to audio timestamps, but not solved the reloading issue.

That was a pretty persistent problem for me too, even after I’d taken steps to prevent it – however, on a hunch, I realized I could overcome it simply by transcluding the video: it reloads each time the src tiddler is modified, so when the timestamp field of the src updates, there’s no way around it – in retrospect, I realize that’s because the way tiddlers are “modified” is that they are actually deleted and then replaced by a tiddler with the original content+modifications, so each time it seemed that the element was reloading, it was actually just losing access to the src for that short period of time.

I’ve got the video and audio parsers both working, I’ll post updates here and to github tomorrow

I’d be interested in seeing your hack.

[etardiff]

@well-noted

I’d be interested in seeing your hack.

It’s not particularly sophisticated — partly because I wasn’t sure how to jump to timestamps other than by modifying the <audio src=>. My template is designed to display one audio file per tiddler, with a link/local file path stored in the audio field. The text field is reserved for timestamped notes (so they’ll show up in standard searches).

In “view” mode, clicking a (green) timestamp takes you to the appropriate point (but reloads the audio element in the process). There’s no current mechanism for jumping back/ahead or saving the current timestamp; I hadn’t figured out how to retrieve that information.

Inline edit mode (modifying the text field of the current tiddler):

The template contains a few global functions/macros (defined elsewhere), but I think it’s probably self-explanatory enough:

\function timestamp() $:/state/timestamp/ [<currentTiddler>] +[join[]]
\function time() [<timestamp>get[text]] ~0 
\function src() [{!!audio}] #t= [<time>] +[join[]]

\function get.seg(n) [<start>split[:]nth<n>trim:prefix[0]]
\function jump.to() [get.seg[1]multiply[3600]] [get.seg[2]multiply[60]] [get.seg[3]] +[sum[]]

\define jump() <$action-setfield $tiddler=<<timestamp>> text=<<jump.to>> />

\define edit()
@@.tc-tiny-gap-left
<% if [{!!editing}!match[]] %>
	<<done-button>>
<% else %>
	<<edit-button>>
<% endif %>
@@
\end

!!! Timestamps <<edit>>
<div class="timestamps-list">
<% if [{!!editing}!match[]] %>
	<<edit-textarea text
			btnClass:"hide"
			placeholder:"""00:05:08 - Topic
00:15:20 - Another topic
01:32:54 - Yet another topic
""">>
<% else %>
<$list filter="[split.line{!!text}!match[]trim[ ]]" variable="line">
<$let
	start={{{ [<line>split[ - ]first[]] }}}
	label={{{ [<line>split[ - ]butfirst[]join[ - ]] }}}
>
	@@.txt-r
		<$button actions=<<jump>> class="tc-btn-invisible tc-tiddlylink">
			<<start>>
		</$button>
	@@
	@@.e <<label>>@@
</$let>
</$list>
<% endif %>
</div>
<div class="audio-player">
<audio controls
	preload=metadata
	src=<<src>>
/>
</div>

[arunnbabu81]

@well-noted two things in which I am interested are

1. To know the current playtime of each video (html videos embedded in separate tiddlers) so that I can do annotation easily.
2. A way to fix the video refresh issues

[well-noted]

New version of the video parser committed to PR, you can see in this preview that it significantly improves upon the version previewed yesterday:

Additional note, one will need to transclude the src tiddler into another tiddler for this to work, because if one is viewing the video within the src, then, when the timestamp updates, the src is temporarily interrupted – so if this were to merged, therefore, we would need to make additional changes so that, if the current tiddler is the src tiddler, it would not store timestamps. Personally, this is a small compromise for me, so I think it reasonable to use right away, in which case you would want to replace the core videoparser with this latest version of the “improved parser”

[well-noted]

@arunnbabu81

@well-noted two things in which I am interested are

1. To know the current playtime of each video (html videos embedded in separate tiddlers) so that I can do annotation easily.
2. A way to fix the video refresh issues

Nothing I’ve done will have any effect on iframe videos – I had thought to commit some time to that, since iframes have been traditionally how I’ve handled this, however, as I’ve briefly noted earlier, I’ve become distrustful of video hosts (Youtube) to preserve unadulterated content from the original – therefore I archive what I want to reference, and knowing MWS was on the way, was waiting to do something along these lines.

If I were to attempt something with youtube links, I would probably attempt to retrieve and store the timestamp as reference values within individual note tiddlers, and then have some button that would modify a variable within the construction of the iframe (which might get very complicated, it certainly sounds like a headache, though I admit it may be my personal biases).

[arunnbabu81]

I was not talking about iframe videos, but html video tag

<video controls src="" type="video/mp4">

I am using a viewtemplate to display the html video tag with src value stored in a tiddler field

[well-noted]

@etardiff, on some reflection, I think the audioparser will need to be a plugin because it includes code that is necessary to make the audiocontrols work too – unless there were some interest in merging the audiocontrols overlay into the core

Was going to make a pretty package, but then realized I will need to make some file changes for that, so in the meantime, you’ll need

• audioparser.js
• audiocontrols.js

You’ll also this CSS stylesheet if you decide to use the audiocontrols
ImprovedAudioParser.css.tid (3.3 KB)

–I believe that the parser will work without the audiocontrols (just don’t include that file in the parsers folder) but there might be dependencies. If you decide to try, let me know – I could probably do a refactoring and/or add the ability to toggle the control overlay.

Per usual, lmk if you encounter any other problems and I’ll try to address in a timely manner

[well-noted]

@arunnbabu81

• To know the current playtime of each video (html videos embedded in separate tiddlers) so that I can do annotation easily.
• A way to fix the video refresh issues

Lmk if the recent commit gives you the tools you need, and what additional features would be helpful. I haven’t begun working on the annotation part of the tool yet, but it is high on my priorities list, so hearing what your requirements are now might be foundational to me thinking.

[arunnbabu81]

This is a demo I had previously shared - it is a wikitext based basic annotation template (it was based on a similar work by telumire and buggyj). What was missing in that concept was the ability to add start and end time of a annotation segment easily since the current playtime value is not stored anywhere. Refresh issues was the second problem i encountered. I didn’t get time to test your updated video parser. I will check it once I am back home. It would have been easy, if there was a demo wiki.

[etardiff]

Interesting, thanks for sharing! I don’t think it will fit into my workflow (since I haven’t been making audio-type tiddlers), but I tested it on TW-com and the controls look quite slick.

I’m not sure how I feel conceptually about the timestamps being stored as fields on the host tiddler itself. TW tends to use $:/state tiddlers to store UI interactions (like unfolding a section of a TOC macro, for instance), and this feels like a comparable situation.

• IMO, playing an audio/video file shouldn’t “dirty” the parent tiddler; I should be able to import it into a different wiki without saving (potentially, someone else’s) timestamps.
• I don’t love the qualified field names, either; they seem likely to result in a lot of field pollution. Using state tiddlers would let you keep the timestamp in the text field(s) and avoid this issue.

The floating player is a nice touch! I do find myself wishing that I could scrub through it as you can with the embedded players, but I don’t know whether that’s feasible, since you’re already using click-and-drag to reposition it in the parent window.

[well-noted]

@etardiff

I do find myself wishing that I could scrub through it as you can with the embedded players

I think it’s possible, it’s definitely something I’ve been considering, but I wanted to finish cleaning up the videoparser, incorporating some of the things I’d learned from building out the audioparser.

The floating player was actually inspired as a mobile feature, as I sometimes need to pause and skip back and forth when taking a note, and within that context:

• it works fairly well, but I find it doesn’t consistently play flawlessly well with all screen interactions (opening context menus in streams, having keyboard open, etc).

• I also think I will not have scrubbing through in the floating window, since (at least on android) those capabilities should be in the notification window

Dragging was an afterthought once I saw how well it seemed to work on desktop, when testing – and, I agree with your assessment that, in that context, scrubbing would be useful as well.

@etardiff

TW tends to use $:/state tiddlers to store UI interactions (like unfolding a section of a TOC macro, for instance), and this feels like a comparable situation

I think you may be right, the state mechanism and the qualify macro are both areas I could use more work in, so perhaps this is the time – it is also a bit more complex, which is why I went with field values initially, but perhaps you could help me work through this conceptually?

• The state would need to be unique both to the source and the context, i.e. $:/state/src.instance

• Notes that are made referencing the “current time” would need to capture that from the state tiddler… so we would need (in plain language, to start) a way to describe the relationship between the context in which that create new note button appears and the $:/state - - so conceptually, is there a way to do that within a template that audio tiddlers are passed through, or would this create new note button need to be a part of the audio parser

• Would you agree that it would be appropriate for timecodes in notes to be field values, since they are not “states” but are static values (or ranges?) In which case, how should the time be called?

1. in the context of the $:/state/src.instance, which would directly tie the tiddler to the instance (and context) of the material in which it was generated,
2. the src itself, which would require altering the state for the src before calling it,
3. within the context of the note itself, i.e, an embedded player in each note (not my favorite solution, but may work if we have multiple template we’re passing through, or
4. some completely different thing I haven’t considered yet?

[well-noted]

@arunnbabu81

What was missing in that concept was the ability to add start and end time of a annotation segment easily since the current playtime value is not stored anywhere.

Ah, yes. The current setup would only allow a “start time” but it has occurred to me to store notes as a range – the tricky thing is that for me, conceptually, recording the note should be as simple as possible… the ideal being click the button, take the note though something where I could drag-select a range, take the note could also be reasonable (though more complex to execute well).

The course you take would be too complicated for me, manually entering multiple edit boxes… here’s another riff, though, what about push button to record start time AND/IF initiate transcription of currently playing, push button to record end time AND/IF suspend transcription, take note?