Playing with Parsers

@buggyj, in response to your question on github

vivaldi_5dRJIm1rID-ezgif.com-resize

Changes I’ve made to the parser allow an embedded or canonical video to buffer and jump to multiple timestamps.

You can see in this example that there is a brief loading period whenever the time is paused or the time changes – that is because the element is reloading each time the timestamp field of the tiddler changes: however, since working on that, I have figured out how to bypass the reloading while working on the audioparser, and I believe I should be able to use the same strategy for the videoparser.

I’d love to see your work on this (either as a plugin or as a core feature, TBH). I’d hacked together a simple interface that lets me skip to audio timestamps, but not solved the reloading issue. Everything you’ve mentioned here sounds like a great QOL improvement!

1 Like

That was a pretty persistent problem for me too, even after I’d taken steps to prevent it – however, on a hunch, I realized I could overcome it simply by transcluding the video: it reloads each time the src tiddler is modified, so when the timestamp field of the src updates, there’s no way around it – in retrospect, I realize that’s because the way tiddlers are “modified” is that they are actually deleted and then replaced by a tiddler with the original content+modifications, so each time it seemed that the element was reloading, it was actually just losing access to the src for that short period of time.

I’ve got the video and audio parsers both working, I’ll post updates here and to github tomorrow :slight_smile:

I’d be interested in seeing your hack.

It’s not particularly sophisticated — partly because I wasn’t sure how to jump to timestamps other than by modifying the <audio src=>. My template is designed to display one audio file per tiddler, with a link/local file path stored in the audio field. The text field is reserved for timestamped notes (so they’ll show up in standard searches).

In “view” mode, clicking a (green) timestamp takes you to the appropriate point (but reloads the audio element in the process). There’s no current mechanism for jumping back/ahead or saving the current timestamp; I hadn’t figured out how to retrieve that information.

Inline edit mode (modifying the text field of the current tiddler):

The template contains a few global functions/macros (defined elsewhere), but I think it’s probably self-explanatory enough:

\function timestamp() $:/state/timestamp/ [<currentTiddler>] +[join[]]
\function time() [<timestamp>get[text]] ~0 
\function src() [{!!audio}] #t= [<time>] +[join[]]

\function get.seg(n) [<start>split[:]nth<n>trim:prefix[0]]
\function jump.to() [get.seg[1]multiply[3600]] [get.seg[2]multiply[60]] [get.seg[3]] +[sum[]]

\define jump() <$action-setfield $tiddler=<<timestamp>> text=<<jump.to>> />

\define edit()
@@.tc-tiny-gap-left
<% if [{!!editing}!match[]] %>
	<<done-button>>
<% else %>
	<<edit-button>>
<% endif %>
@@
\end

!!! Timestamps <<edit>>
<div class="timestamps-list">
<% if [{!!editing}!match[]] %>
	<<edit-textarea text
			btnClass:"hide"
			placeholder:"""00:05:08 - Topic
00:15:20 - Another topic
01:32:54 - Yet another topic
""">>
<% else %>
<$list filter="[split.line{!!text}!match[]trim[ ]]" variable="line">
<$let
	start={{{ [<line>split[ - ]first[]] }}}
	label={{{ [<line>split[ - ]butfirst[]join[ - ]] }}}
>
	@@.txt-r
		<$button actions=<<jump>> class="tc-btn-invisible tc-tiddlylink">
			<<start>>
		</$button>
	@@
	@@.e <<label>>@@
</$let>
</$list>
<% endif %>
</div>
<div class="audio-player">
<audio controls
	preload=metadata
	src=<<src>>
/>
</div>

@well-noted two things in which I am interested are

  1. To know the current playtime of each video (html videos embedded in separate tiddlers) so that I can do annotation easily.
  2. A way to fix the video refresh issues

New version of the video parser committed to PR, you can see in this preview that it significantly improves upon the version previewed yesterday:

vivaldi_J7wOsRNP5b-ezgif.com-resize

Additional note, one will need to transclude the src tiddler into another tiddler for this to work, because if one is viewing the video within the src, then, when the timestamp updates, the src is temporarily interrupted – so if this were to merged, therefore, we would need to make additional changes so that, if the current tiddler is the src tiddler, it would not store timestamps. Personally, this is a small compromise for me, so I think it reasonable to use right away, in which case you would want to replace the core videoparser with this latest version of the “improved parser”

Nothing I’ve done will have any effect on iframe videos – I had thought to commit some time to that, since iframes have been traditionally how I’ve handled this, however, as I’ve briefly noted earlier, I’ve become distrustful of video hosts (Youtube) to preserve unadulterated content from the original – therefore I archive what I want to reference, and knowing MWS was on the way, was waiting to do something along these lines.

If I were to attempt something with youtube links, I would probably attempt to retrieve and store the timestamp as reference values within individual note tiddlers, and then have some button that would modify a variable within the construction of the iframe (which might get very complicated, it certainly sounds like a headache, though I admit it may be my personal biases).

I was not talking about iframe videos, but html video tag

<video controls src="" type="video/mp4">

I am using a viewtemplate to display the html video tag with src value stored in a tiddler field

@etardiff, on some reflection, I think the audioparser will need to be a plugin because it includes code that is necessary to make the audiocontrols work too – unless there were some interest in merging the audiocontrols overlay into the core :man_shrugging:

Was going to make a pretty package, but then realized I will need to make some file changes for that, so in the meantime, you’ll need

You’ll also this CSS stylesheet if you decide to use the audiocontrols
ImprovedAudioParser.css.tid (3.3 KB)

–I believe that the parser will work without the audiocontrols (just don’t include that file in the parsers folder) but there might be dependencies. If you decide to try, let me know – I could probably do a refactoring and/or add the ability to toggle the control overlay.

Per usual, lmk if you encounter any other problems and I’ll try to address in a timely manner

1 Like

Lmk if the recent commit gives you the tools you need, and what additional features would be helpful. I haven’t begun working on the annotation part of the tool yet, but it is high on my priorities list, so hearing what your requirements are now might be foundational to me thinking.

This is a demo I had previously shared - it is a wikitext based basic annotation template (it was based on a similar work by telumire and buggyj). What was missing in that concept was the ability to add start and end time of a annotation segment easily since the current playtime value is not stored anywhere. Refresh issues was the second problem i encountered. I didn’t get time to test your updated video parser. I will check it once I am back home. It would have been easy, if there was a demo wiki.

Interesting, thanks for sharing! I don’t think it will fit into my workflow (since I haven’t been making audio-type tiddlers), but I tested it on TW-com and the controls look quite slick.

I’m not sure how I feel conceptually about the timestamps being stored as fields on the host tiddler itself. TW tends to use $:/state tiddlers to store UI interactions (like unfolding a section of a TOC macro, for instance), and this feels like a comparable situation.

  • IMO, playing an audio/video file shouldn’t “dirty” the parent tiddler; I should be able to import it into a different wiki without saving (potentially, someone else’s) timestamps.
  • I don’t love the qualified field names, either; they seem likely to result in a lot of field pollution. Using state tiddlers would let you keep the timestamp in the text field(s) and avoid this issue.

The floating player is a nice touch! I do find myself wishing that I could scrub through it as you can with the embedded players, but I don’t know whether that’s feasible, since you’re already using click-and-drag to reposition it in the parent window.

I think it’s possible, it’s definitely something I’ve been considering, but I wanted to finish cleaning up the videoparser, incorporating some of the things I’d learned from building out the audioparser.

The floating player was actually inspired as a mobile feature, as I sometimes need to pause and skip back and forth when taking a note, and within that context:

  • it works fairly well, but I find it doesn’t consistently play flawlessly well with all screen interactions (opening context menus in streams, having keyboard open, etc).

  • I also think I will not have scrubbing through in the floating window, since (at least on android) those capabilities should be in the notification window

Dragging was an afterthought once I saw how well it seemed to work on desktop, when testing – and, I agree with your assessment that, in that context, scrubbing would be useful as well.

I think you may be right, the state mechanism and the qualify macro are both areas I could use more work in, so perhaps this is the time – it is also a bit more complex, which is why I went with field values initially, but perhaps you could help me work through this conceptually?

  • The state would need to be unique both to the source and the context, i.e. $:/state/src.instance

  • Notes that are made referencing the “current time” would need to capture that from the state tiddler… so we would need (in plain language, to start) a way to describe the relationship between the context in which that create new note button appears and the $:/state - - so conceptually, is there a way to do that within a template that audio tiddlers are passed through, or would this create new note button need to be a part of the audio parser

  • Would you agree that it would be appropriate for timecodes in notes to be field values, since they are not “states” but are static values (or ranges?) In which case, how should the time be called?

  1. in the context of the $:/state/src.instance, which would directly tie the tiddler to the instance (and context) of the material in which it was generated,
  2. the src itself, which would require altering the state for the src before calling it,
  3. within the context of the note itself, i.e, an embedded player in each note (not my favorite solution, but may work if we have multiple template we’re passing through, or
  4. some completely different thing I haven’t considered yet?

Ah, yes. The current setup would only allow a “start time” but it has occurred to me to store notes as a range – the tricky thing is that for me, conceptually, recording the note should be as simple as possible… the ideal being click the button, take the note though something where I could drag-select a range, take the note could also be reasonable (though more complex to execute well).

The course you take would be too complicated for me, manually entering multiple edit boxes… here’s another riff, though, what about push button to record start time AND/IF initiate transcription of currently playing, push button to record end time AND/IF suspend transcription, take note?

https://mediaplayerava.tiddlyhost.com/

My annotation template was inspired by this mediaplayer by @buggyj

This should also be enough.

Other use of the mediaplay is here

https://mediaplayers.tiddlyhost.com/

can you share a screenshot of how your video tiddler looks like so that I can test the code from the PR

It’s certainly possible to do it with a template (in the traditional TW sense). I’m not familiar with the parsers and the ways they intersect with content-types… You could use the ViewTemplate cascade to supply a default template when rendering any tiddler with an audio/... type, but this wouldn’t help you when audio/video tiddlers get transcluded into other contexts, so I suspect you’d need to modify the parser if you wanted an annotation mechanism to be universally available.

Broadly speaking, though, qualify works the same way whether you’re using it to generate a field name or a tiddler title – so if you can retrieve the value from a qualified field name, you can retrieve it from the text field of a qualified $:/state tiddler.

Yes, I’d say time-stamped notes do fall into the category of “content”, not “state”. I’m not sure how likely it is that you’d want different notes/timestamps on the same media in different contexts, but I can contrive some possible circumstances — for instance, I might want one set of notes on the topics discussed, and another on linguistic phenomena displayed. And as I said previously, I’m not keen on the idea of “dirtying” the src tiddler with fields that won’t be meaningful if taken out of context, since TW users do commonly import tiddlers from other wikis.

  • I’d generally prefer to keep any timecode notes on the tiddler in which the player was embedded, since that’s presumably the context in which they’re most relevant.
    • I think my personal ideal would be one note field per embedded player per host tiddler, with each timecode and its associated note appearing on a new line (akin to a dictionary tiddler, but stored in a field).
      • In my own (very simple, manually-edited) template, I’m using the text field this way, but a) I only have one embedded player per tiddler, and b) I’m not using the text fields of those tiddlers for anything else. For “playlist” tiddlers, you’d presumably want a different field for each player.
  • I could also see a case for storing notes in JSON tiddlers associated with the “host” tiddler.
    • This would avoid the potential field pollution issue altogether, and it would let you use just one “notes” tiddler per “host” tiddler, regardless of the number of embedded players.
    • Of course, a secondary “notes” tiddler would be less closely linked to both the media src tiddler and the “host” tiddler displaying it. This could be either a downside or a benefit, depending on your perspective.
      • On the one hand, it’s an extra tiddler to import if I want the same notes in another wiki.
      • On the other hand, perhaps I want your video collection but not your annotations?
    • The other, larger downside to JSON tiddlers is that reading and especially writing to them is still rather difficult, IMO. You’d need a fully automated writing/retrieval system that wouldn’t require any end-user knowledge of what was going on behind the scenes.

I do think we’re getting rather far into the territory of personal preference here, and I’m not convinced that there is one “optimal” solution. Other people may want to store/access/display their notes in ways I haven’t considered, and this may be too opinionated to hard-code into a parser.

I think it’d be more in-line with the general TW ethos to keep the parser/player itself as simple and flexible as possible and use procedures or widgets to provide more complicated systems like annotations. This would let you use {{My Video}} to display just the player and <<annotated-video "My Video">> to display a template that generates/displays your timecode notes.

  • This is basically the approach @buggyj takes in his (very impressive!) media player plugins… though I admit that they’re so complex I found them a little intimidating to use.

I’m sorry I don’t quite understand what you’re asking here. I included a gif above that shows what the transclusion looks like when the parser has been replaced, and the code for the tiddler is just a transclusion of the src tiddler {{billionaires want you to know they could have done physics.mp4}}

Well, let’s just say we have a tag “Audio” or something, for simplicity’s sake – what would the plain-english way of describing the relationship between the note and the context be? For example, "Create a new tiddler which has a field X with the value <<currentTime>>" x being some value that would communicate both the parent tiddler (The one tagged audio, transcluding the src) and also the particular audioplayer instance (in case you are transcluding multiple audio files in the same Audio tiddler)

Agreed, I’d like to find a generic enough solution that will work for a wide range of usecases, hence my asking so many questions (forgive :slight_smile: )

I use Streams for notes, so each note is its own tiddler which is being transcluded in the stream-list of the parent tiddler. This allows me to transclude notes in multiple place – that’s helpful because, in some contexts, notes are being used in a processing/reference context, and others in a reference/citation context, which I am constructing an argument which might be supported by evidence the note points to.