Parsing text into content nested under reveal widget

[well-noted]

Seems I’ve been working a lot on text parsing recently… I’m working on a survey template that uses sections marked with ! prefixes to organize questions.

While I can successfully identify and display the section headers, I’m having trouble capturing the content between these section markers within the reveal widget.

Here’s the relevant part of the code that’s not working as expected:

<$reveal type="nomatch" state=<<section-state>> text="hidden" default="">
    <div class="section-content">
        <$list filter="[<currentTiddler>get[survey-text]split[

]!prefix[!]] :filter[<currentTiddler>get[survey-text]split[

]prefix[!]first<firstLine>last[]removesuffix<currentBlock>]" variable="question">
            <!-- Survey item content here -->
        </$list>
    </div>
</$reveal>

When I use similar filtering logic outside the reveal widget for debugging, it works correctly:

<!-- Debug code that works -->
<$vars currentSection="">
    <$list filter="[<currentTiddler>get[survey-text]split[

]]" variable="line">
        <$list filter="[<line>prefix[!]]" variable="newSection">
            <div class="debug-section-start">
                New Section: <$text text={{{ [<newSection>removeprefix[!]trim[]] }}}/>
            </div>
            <$action-setfield _currentSection=<<newSection>>/>
        </$list>
        
        <$list filter="[<line>!prefix[!]!is[blank]]" variable="content">
            <div class="debug-content-line">
                Content for section <$text text={{{ [<_currentSection>removeprefix[!]trim[]] }}}/>:
                • <$text text=<<content>>/>
            </div>
        </$list>
    </$list>
</div>

I’ve tried several approaches:

1. Using state variables to store section content
2. Creating a separate macro for content processing
3. Modifying the filter chains

But, while the reveal state works for the section headers, I cannot render the desired content as nested inside it. I suspect it might be related to the filter context within the reveal widget? Toying with it the last few days, I haven’t been able to budge it.

Sample input text:

!test

Question 1

Question 2

!another section

Question 3
a description

Question 4

Any takers?

[Springer]

I don’t have time to dig deep on the details.

I’ll say off the cuff, though, that putting line breaks “inline” within long code has been fragile for me.

Better to define a variable — the set widget can be tagged $:/tags/Global — and get into the habit of using that variable:

<$set name="doublebreak" value="

"/>

This allows you to reference <doublebreak> anywhere, without needing to have linebreaks interrupting your more complex expressions.

[EricShulman]

While a global definition for doublebreak is a good idea, there’s two issues with the code suggested by @springer:

• Using $:/tags/Global will not invoke the $set widget to define a global variable. Rather, it will only process \define, \procedure, and \function definitions.

• Rather than embedding literal newlines into the code, I recommend using the charcode[...] operator to generate the appropriate linefeed characters (ASCII code 10).

The following global function definition will work:

\function double.break() [charcode[10],[10]]

which can then be referenced as <<double.break>> (or <double.break> when used in filter syntax).

-e

[Springer]

I’m confused…

@EricShulman

• Using $:/tags/Global will not invoke the $set widget to define a global variable. Rather, it will only process \define, \procedure, and \function definitions.

… I’m looking at tw-com, which says…

Using the Set Widget to Create Global Variables

There are times when it makes sense to use the features of the SetWidget rather than procedures or functions to create global variables. This can be accomplished by placing the set variable widget in a tiddler that is tagged $:/tags/Global. …

At any rate, I’m always happy to defer to you about best practices!

[well-noted]

@EricShulman

Rather than embedding literal newlines into the code, I recommend using the charcode[...] operator to generate the appropriate linefeed characters (ASCII code 10).

Clever idea, a good way to clean things up – however, replacing the literal with the charcodes works exactly the same for me

[etardiff]

1. How are you defining <<section-state>>? If you add some hard-coded content in the section-content div (outside the $list), does it get properly displayed?

2. Since you’re not animating the $reveal anyway, as an additional debug step I’d try replacing

<$reveal type="nomatch" state=<<section-state>> text="hidden" default="">

...
</$reveal>

with

<% if [<section-state>!text[hidden]] %>

...
<% endif %>

[well-noted]

@etardiff

How are you defining <<section-state>>? If you add some hard-coded content in the section-content div (outside the $list), does it get properly displayed?

\define section-state() $:/state/$(currentTiddler)$/section/$(sectionId)$

Hard coded content does display, (with both reveal and if/endif, you’re always catching me on this one @etardiff )

I assume the problem must therefore be with the content filter:

<$list filter="[<currentTiddler>get[survey-text]split<double.break>!prefix[!]] :filter[<currentTiddler>get[survey-text]split<double.break>prefix[!]first<firstLine>last[]removesuffix<currentBlock>]" variable="question">

[etardiff]

@well-noted

[<currentTiddler>get[survey-text]split<double.break>!prefix[!]] :filter[<currentTiddler>get[survey-text]split<double.break>prefix[!]first<firstLine>last[]removesuffix<currentBlock>]

I admit I haven’t studied your code very closely, but is it possible that :filter[<currentTiddler>... should be :filter[<..currentTiddler>? As written, you’re getting the survey-text field from the tiddler whose title corresponds to the output of the previous filter run… which is possible, but seems like it might not be your intent…

[well-noted]

@etardiff

@well-noted

[<currentTiddler>get[survey-text]split<double.break>!prefix[!]] :filter[<currentTiddler>get[survey-text]split<double.break>prefix[!]first<firstLine>last[]removesuffix<currentBlock>]

I admit I haven’t studied your code very closely, but is it possible that :filter[<currentTiddler>... should be :filter[<..currentTiddler>? As written, you’re getting the survey-text field from the tiddler that corresponds to the output of the previous filter run… which is possible, but seems like it might not be your intent…

I’ve never used that <..currentTiddler> before, good to know about – but didn’t fix the issue.

EDIT, or is this not a real macro, but a hypothetical one?

Outside of any of these lists, I am capable of getting the section heads and the content appropriate to those sections listed with this

<$list filter="[all[current]tag[Survey]]">
<div class="survey-container">
    <!-- Debug: Section Content Analysis -->
    <div style="background: #fff3cd; padding: 15px; margin: 10px 0; border: 1px solid #ffeeba;">
        <h4>Debug: Section Content Analysis</h4>
        
        <$vars currentSection="">
            <!-- First show all sections -->
            <div style="margin-bottom: 20px;">
                <strong>All Sections Found:</strong>
                <$list filter="[<currentTiddler>get[survey-text]split<double.break>prefix[!]]" variable="section" counter="sectionNum">
                    <div style="margin-left: 20px; color: #060;">
                        Section <<sectionNum>>: <$text text=<<section>>/>
                    </div>
                </$list>
            </div>

            <!-- Now show sections with their content -->
            <div style="margin-top: 20px;">
                <strong>Sections with Content:</strong>
                <$list filter="[<currentTiddler>get[survey-text]split<double.break>]" variable="line">
                    <!-- Section headers -->
                    <$list filter="[<line>prefix[!]]" variable="section">
                        <div style="margin-top: 15px; padding: 5px; background: #e0e0e0;">
                            <strong>New Section: <$text text=<<section>>/></strong>
                            <$action-setfield _currentSection=<<section>>/>
                        </div>
                    </$list>
                    
                    <!-- Content lines -->
                    <$list filter="[<line>!prefix[!]!is[blank]]" variable="content">
                        <div style="margin-left: 20px; padding: 3px; color: #600;">
                            • <$text text=<<content>>/>
                            <br/>
                            <small style="color: #666;">(Under section: <$text text=<<_currentSection>>/>)</small>
                        </div>
                    </$list>
                </$list>
            </div>
        </$vars>
    </div>

EDIT: have confirmed, we are losing context inside of the list.

[etardiff]

It’s a bit difficult to debug without access to the rest of your testing environment — for instance, you’re using the variables <<firstLine>> and <<currentBlock>>, but they’re not defined in the context you’ve provided. So I’ll just offer my generalized approach to filter debugging, which is to paste your $list variable (<<question>>, in this case) inside the $list, where you currently have the survey item content. Then either start removing segments of the filter until you get the results you expect, or start from a simpler form of the filter and re-add operators until it breaks.

[well-noted]

Sorry, what’s the demo site maker I see people use sometimes? I can’t seem to find it with a google or forum search.

[etardiff]

tiddlyhost.com (plus extra text to meet the minimum character threshold)

[well-noted]

lawl, brainfart, silly me.

My TiddlyWiki

[etardiff]

The first couple of things I’m noticing:

1. You don’t need to define <<..currentTiddler>>; this is a special variable defined within the context of some prefixed filter runs, like :filter and :map, which redefine <<currentTiddler>> to refer to each input value of that run. <<..currentTiddler>> preserves the original value of <<currentTiddler>> outside the filter.

<$list filter="[<currentTiddler>get[survey-text]split<double.break>]" variable="block">
    <$let 
        firstLine={{{ [<block>split<single.break>first[]] }}}
        description={{{ [<block>split<single.break>rest[]join[ ]] }}}
    >
        <$list filter="[<firstLine>prefix[!]]">

2. Here, you’re splitting blocks at double line breaks, with the first line becoming the header. But your sample text also uses double line breaks after each h1, e.g.

! Section 2 

This is another section

This means that “This is another section” becomes the <<firstLine>> of its own block, rather than the <<description>> of Section 2 — but since it’s not prefixed with !, it’s not displayed by the following list. And conversely, Section 2 has no content to display even when it’s unfolded because there are no other lines in the same block.

[well-noted]

I see, I was confused about ``<<…currentTiddler>> That’s very good info to know. I thought I might as well redefine it so I’d know it had the correct value.

This is another section to represent a question, without a description.

This is what I’m going for, but as you can see from the expanded/collapsed state, the question is not properly nested.

That the firstline is becoming redefined, though, is relevant…

[VikingMage]

I think this is what you are after.

\function example-tiddler-content()[[\n!test\n\nQuestion 1\n\nQuestion 2\n\n!another section\n\nQuestion 3\na description\n\nQuestion 4]search-replace:g[\n],<lf>]
\function lf()[charcode[10]]
\function lfx2()[charcode[10],[10]]

<$list filter="[<example-tiddler-content>splitregexp[\n\n(?=!)]trim[]]" variable=section>

<$let
  heading={{{ [<section>split<lf>prefix[!]] }}}
  sectionIndex={{{ [<heading>trim[!]] }}}
  questionStateTid="$:/state/surveyquestions"
>

<$button>

  <<heading>>
  <$action-setfield
    $tiddler=<<questionStateTid>>
    $index=<<sectionIndex>>
    $value={{{ [<questionStateTid>getindex<sectionIndex>match[hide]then[show]] ~[[hide]] }}}
  />
</$button>

<$reveal
  type="match"
  stateTitle=<<questionStateTid>>
  stateIndex=<<sectionIndex>>
  text="show"
  default="show"
>

<div class="section-content">

<$list
  filter="[<section>removeprefix<heading>trim[]split<lfx2>]"
  variable=question
>

<$let
  questionHeading={{{ [<question>split<lf>first[]] }}}
  questionDescription={{{ [<question>removeprefix<questionHeading>trim[]] }}}
  questionIndex={{{ [<sectionndex>addsuffix<questionHeading>]  }}}
>

!!! <<questionHeading>>
    ><<questionDescription>>

</$let>
</$list>
</div>
</$reveal>
</$let>
</$list>

[well-noted]

Yep, that did it Thank you, @VikingMage, this had me wracking my brains the last couple of days.

For anyone who wants to see the exact implementation I used:

\function double.break() [charcode[10],[10]]
\function single.break() [charcode[10]]

\define survey-results()
Results of '$(currentTiddler)$' taken on <<now "YYYY-0MM-0DD 0hh:0mm UTC">>

<$list filter="[<currentTiddler>fields[]prefix[response-]]">
Question: <$text text={{{ [<currentTiddler>] }}} />
Answer: <$text text={{{ [<currentTiddler>get<currentTiddler>] }}} />

</$list>
\end

\function f.opFields() [fields[]prefix[response]] -[[text]] -[[title]]

<$list filter="[all[current]tag[Survey]]">

<div class="survey-container">


<$list filter="[<currentTiddler>get[survey-text]splitregexp[\n\n(?=!)]]" variable="section">
<$let
  heading={{{ [<section>split<single.break>prefix[!]] }}}
  sectionIndex={{{ [<heading>trim[!]] }}}
  questionStateTid="$:/state/surveyquestions"
>

<div class="section-header">
    <$button class="tc-btn-invisible section-toggle">
        <$text text={{{ [<heading>trim[!]] }}}/>
        <$action-setfield
            $tiddler=<<questionStateTid>>
            $index=<<sectionIndex>>
            $value={{{ [<questionStateTid>getindex<sectionIndex>match[hide]then[show]] ~[[hide]] }}}
        />
    </$button>
</div>

<$reveal
    type="match"
    stateTitle=<<questionStateTid>>
    stateIndex=<<sectionIndex>>
    text="show"
    default="hide"
>

<div class="section-content">
    <$list
        filter="[<section>removeprefix<heading>trim[]split<double.break>]"
        variable="question"
    >
    <$let
        questionHeading={{{ [<question>split<single.break>first[]] }}}
        questionDescription={{{ [<question>removeprefix<questionHeading>trim[]] }}}
        questionIndex={{{ [<sectionIndex>addsuffix<questionHeading>] }}}
    >
    
    <div class="survey-item">
        <div class="question">
            <strong><$text text=<<questionHeading>>/></strong>
            <$list filter="[<questionDescription>!is[blank]]">
                <div class="description"><em><$text text=<<questionDescription>>/></em></div>
            </$list>
        </div>
        <div class="rating-bubbles">
            <$radio tiddler=<<currentTiddler>> 
                field={{{ [<questionHeading>addprefix[response-]] }}} 
                value="1"
                default="0">
                <span class="bubble-option">1</span>
            </$radio>
            <$radio tiddler=<<currentTiddler>> 
                field={{{ [<questionHeading>addprefix[response-]] }}} 
                value="2"
                default="0">
                <span class="bubble-option">2</span>
            </$radio>
            <$radio tiddler=<<currentTiddler>> 
                field={{{ [<questionHeading>addprefix[response-]] }}} 
                value="3"
                default="0">
                <span class="bubble-option">3</span>
            </$radio>
            <$radio tiddler=<<currentTiddler>> 
                field={{{ [<questionHeading>addprefix[response-]] }}} 
                value="4"
                default="0">
                <span class="bubble-option">4</span>
            </$radio>
            <$radio tiddler=<<currentTiddler>> 
                field={{{ [<questionHeading>addprefix[response-]] }}} 
                value="5"
                default="0">
                <span class="bubble-option">5</span>
            </$radio>
        </div>
        <div class="rating-labels">
            <span>Not at all</span>
            <span>Yes, absolutely</span>
        </div>
    </div>

    </$let>
    </$list>
</div>
</$reveal>
</$let>
</$list>

And thanks @etardiff, @Springer, and @EricShulman for contributing essential pieces to bring it all together.

[VikingMage]

welcome, I had been wanting to work out how to do the splitregexp with a look ahead like that for a while. this was a good excuse to figure it out.