Expanded ChatGPT Interface

[well-noted]

Hey all, I posted a while back in ChatGPT for TiddlyWiki to suggest some fixes and improvements - - I have continued to play around with expanding its capabilities and I’ve reached the point where it feels like posting a separate plugin would be appropriate and useful

$__plugins_NoteStreams_Expanded-Chat-GPT.json (46.2 KB)

Please note, this is my first time packaging and releasing a plugin - - I’ve likely missed steps and may not have done enough to acknowledge the original plugins creator (@Sttot) and/or quite a lot of other things, though I have tested this on a blank Wiki and it works. Most of my experience with Tiddlywiki and javascript more generally has come from kit-bashing other people’s projects. I have recently decided to begin repackaging some of the (many) customized versions of plugins that I use, but will need some time to untangle the hairs and perform testing.

Please let me know if there’s anything inappropriate about sharing this plugin or the format of this post and I will attempt to correct.

Also, I value @linonetwo for pointing out in the original ChatGPT post that the new version of TW has a chatbot plugin under development which also has the (exciting!) ability to use local models - - as far as I can tell, that version is currently (and maybe even only intended) to be a chat bot.

That is only one of the features of this plugin, however. An AI agent within Tiddlywiki has been a long time dream for me, and I find that a chatbot, while cool, does not do quite enough to justify itself.

Here are some of the important features of this plugin:

1. Enhanced Memory: The agent can remember 5 previous messages by default, but this can be adjusted in the widget settings.
2. Reference Tiddler: This version creates a reference tiddler to store content deemed important about the user, enhancing long-term memory and personalization.
3. Default Model: Uses gpt-4o-mini by default, which can be changed in the widget however please note that this widget uses multiple API calls at its own discretion in order to complete a request, each one of these costing money
   That said, even though I have been extensively testing using both simple and complex prompts over the last several days, the most I’ve paid on any given day, using this as the default model, was $0.60 and even then I was prompting, modifying the code, and prompting again.
4. Contextual Awareness: The system prompt provides context for the agent, including its TiddlyWiki environment and its capabilities to reference and modify tiddlers.
5. Extended Capabilities: The AI assistant can interact with TiddlyWiki in various ways, including:
   • Retrieving tiddler titles and content
   • Searching tiddlers by content, tags, or custom fields
   • Adding, retrieving, and revising notes about the user
   • Accessing conversation history
   • Creating and modifying tiddlers
6. Streams Plugin Compatibility: By default, this plugin is compatible with the Streams plugin. The agent will refer to the content within a tiddler’s stream-list field in addition to the content of the tiddler itself, providing a more comprehensive context.
7. Specific Tiddler Reference: Users can reference a specific tiddler as context for the conversation by using the tiddlerTitle parameter in the widget.
8. Access to the Current Time: As in, the agent can check the date and report info about tasks that may be overdue, for example, or list tiddlers that were created on a particular day. It can also be used to create timestamps which can be used in text, fields, tags, or titles.

I have not attempted to play with the new versions chatbot yet – though I am incredibly enthusiastic about the changes in the new version, I have not done any work with SQLite as of yet and have no idea if this version I’m releasing would be easily converted to utilize the SQLite database or not. I can report that I developed and tested on a MultiWikiServer instance, though, and it seems to do data retrieval well. That said, it is compatible with both the prerelease and the current version, at the very least.

Alright, now that acknowledgements are over, let’s look at some examples so we can talk about the great advantages that can be leveraged by using ML within a TW Environment.

As you can see, this agent has been given the ability to create new tiddlers based on natural language prompts.

Here you can see a demonstration of 1) its conversational memory and 2) its ability to make changes to existing tiddlers based on the users natural language request.

You can see that the agent is not only able to recognize specific tiddlers and call information from them, but will also consider content from that tiddler’s stream-list when answering a prompt, if it deems that appropriate. It can do this by searching for a matching title or for the content in any of the tiddlers aliases fields

Here you can see that the agent self-selects information that it deems important and stores that in a reference tiddler, which it is able to use when it deems appropriate to responding to a user’s prompt. This allows one to “train” (deceptive choice of word in this context) the agent, by explaining important parts of one’s personal workflow. This is not specifically called out in the above example, but one can see that the agent does not simply add to this list, but can recognize if important information is already in the reference material, and (revise) the entry without modifying the rest of the content.

And I have also personally followed @jeremyruston 's example and used a small brain icon to “summon” the agent – this is not included in the plugin, but is simple enough to implement

It’s also possible to create a sidebar agent which directly references the storyTiddler – you can do this with

<$chat-gpt useCurrentTiddler="yes" />

{{$:/HistoryList!!current-tiddler}} 

The above code is telling the widget to send the contents of the storyTiddler along with the prompts, and then it is transcluding the title below that for the users reference. This way, one might interact with their wiki without having to specify to which tiddler they are referring.

All of this together, I think, creates an incredibly helpful tool for interacting with one’s own content.

Some improvements I’m considering for the future:

• Wikify responses within the chatbox
• Give the agent the ability to open tiddlers in the story river
• Expanded control over tiddlers (i.e. the capability to delete tiddlers – with a user confirmation – from the interface) For obvious reasons, I have been unwilling to add this feature until I was sure the plugin was stable-ish.
• Expanded Reference capability that the agent might categorize its own reference material and to do so in a way that is more legible to the user (if one wants to review what it is the agent is using as reference)
• Multimodal capabilities (image recognition, voice to text/text to voice, etc.)

[linonetwo]

These are features that I also want in core AI tools plugin, maybe you can PR them to the original PR (fork the repo, check out the branch, and create new branch based on that branch. And create PR , select the target branch to be that brahcn).

This way, we can make sure future AI features are on the same foundation. And as you may see, we are deprecating the old chatgpt plugin in favior of the core plugin one. Building new features based on it may not get enough discussion like dev in the core AI plugin.

I don’t have time for AI plugins recently, because I want to refactor the WYSIWYG editor plugin, so AI in it can works like AI in notion.

[Scribs]

great work, this looks super useful!

[CodaCoder]

@well-noted

That’s an extraordinary feat of accomplishment. Well done.

You would do well to mention Safety and Security, how prompts are tunneled, what stays local and what leaves the local machine.

The linked article here concerns me a great deal: LLMs NOT welcome in my wikis -- yet.

[well-noted]

Thank you, @CodaCoder – I’m happy to discuss, with the caveats that 1), this will not be a comprehensive discussion of the topic at this time and 2) I do not use my TW to handle especially sensitive information, i.e. information I wouldn’t be more than mildly embarrassed about (such as personal reflections). It should go without saying that if you handle national security secrets, for example, you shouldn’t include this plugin in your wiki.

^{Minor note on point 2, however, my understanding leads me to believe that one of the most exciting parts of MWS would be that sensitive data kept in one bag could be viewed in the context of a recipe that also includes a general bag full of less sensitive, and the bag containing the sensitive data could also be kept separate from the recipe which includes access from a non-local LLM}

---

To your point, broadly:

When making an API call with OpenAI, inputs are no longer used for training unless one applies for that through a multi-step process. This is different than if one uses the ChatGPT interface in which one has to opt out of training data. There is definitely real concern for the gen-pop user who might be discussing sensitive work data with ChatGPT through the web-interface who may not have gone to the trouble of opting out - - it has been documented time and time again that LLMs are especially sensitive to “coercive behavior” designed to override their usual protocols. That said, I am not especially concerned in this context about data being passed through API calls being accessible via one of these methods of attack.

The particular behavior the article you link to in your post, (Invisible text that AI chatbots understand and humans can’t? Yep, it’s a thing. - Ars Technica) does make clear that OpenAI API calls do not include sensitive information within invisible text and therefore its users are not susceptible to the potential security threats related to that.

This, to my eye, is one of the most important reasons to opt for OpenAI (API calls specifically) at this time (though I do like Claude, for example, for certain tasks): Because they have “all-eyes-on-them” as the big name in ML tech (I don’t include Google because it is so diversified whereas OpenAI is “all-in”) they have an especially strong interest in finding strategies to avoid major security pitfalls such as the invisible text problem described. I have seen @linonetwo suggest Anthropic API calls - - at this time, I would not use Claude for contextual searches such as those enabled by this plugin.

---

Now:

There is a cache of information that is very relevant to this conversation but I have not mentioned yet - - pretty recently, OpenAI began caching common user prompts that are made through API calls. This cache is tied directly to your API key - - if someone had your API key, they could certainly access this cache through coercive prompting.

However, the larger point here is that, if you have structured your prompts well (in this case, you would want the system prompt, which is generally static, to precede the user prompt, which is usually dynamic) the agent will select to pull that information from the cache rather than handling it as original text to process (again, broad overview of the behavior). This means (generally) that, once someone has run the same prompt multiple times, they will avoid having to pay for processing the same system prompt multiple times, and they will only have to pay for the original content.

For example, I use a script that processes all of my reading notes from a plaintext format:

^{Exported from Boox}

Into individual jsons which are formatted for Tiddlywiki and contain all the tags:

^{Individual JSON format created by running python script which reorganizes information via openAI api call to gpt-4o-mini}

I have a pretty detailed (and extensive) prompt which I have developed over time to produce incredibly consistent results using even small models like gpt-4o-mini – The script I’ve developed passes this (static) system prompt through the API call first, and then extracts the characters from a .txt file in order to reprocess them.

Since I do this nearly every day to import notes into my wiki, the system prompt is cached (and tied to my unique API key) so the system recognizes that it can skip over that content – then I am only paying for the dynamic content which are my notes (usually this costs less than a cent a day even for multiple chapters across multiple books).

For our purposes here, it means that we can have a very extensive system prompt – say, one that explains how Tiddlywiki works, how the agent should respond to different situations, a full and detailed list of its capabilities, a set of fallback instructions for handling unique or unfamiliar situations, etc, without the cost becoming excessive in the long run, and, as long as you don’t share your API key could even include personal information.

It is therefore important, when developing a strategy like this, to make sure that all static prompting precedes dynamic prompting

[CodaCoder]

Thank you. Much appreciated.