Unlocking live context in prompts and variables

[einarpersson]

Hi! I am so excited about code companion!

I would like to unlock 'live context capability'. What do I mean with that? Well, a lot of contextual information changes over time, it becomes stale. The most obvious example is the time/clock itself. I would like to include such context in the system prompt and have it refreshed on each new message submitted.

I have used this capability in other ai chat tools with great success, it really makes the user experience much smoother. I made a private fork of aichat just for this purpose. In that case it made my shell assistant aware of the cwd, time, etc.. small things that helped the assistant be more accurate as I made sure it had the current information. Doing it this way was also very economical token-wise. The one thing to be aware of is that I had to wrap the live data in a kind of "live data block" which made it clear to the LLM that this data was always the current one and that this would explain any discrepancies with the following conversation. Without that, there could be some confusion. But with that, it worked really well!

I have been looking at the docs for extending prompts and was glad to find that I could provide functions to generate dynamic prompts. However, I was lacking a live = true or refresh = true that would enable the capability I was talking about above.

Suggestion 1

So one concrete suggestion is to add such a property to the message tables. Then on each new message sent the function should re-run and replace that message in the chat conversation with the same index.

Or it should perhaps be for simplicity be added in the opts-table on the whole prompt. But that would make it less granular and less performant. And I'm also not sure about what it would mean for prompts with references

---

A related note:

I was also fiddling around with #buffer and #buffer{pin}. As noted in the docs, pinning a buffer increases the buffer usage a lot since a whole new copy of the buffer is appended on each roundtrip.

When I first saw the pinning feature, I actually thought that it was more inline with the 'live data' concept above. I.e. that it was inserted/appended to the system prompt and that it was replaced/refreshed on each new user message sent. This is for me a more natural meaning of 'pinning'.

(Note: I'm aware of the watch feature and it may be good, but I would ideally want something like above to be possible.)

And to solve a more general problem, when looking in the docs about creating your own variables I would like something similar as described above, a key such as refresh = true or live = true or reevaluate = true. It could be injected into the system prompt with a <live-context>-block or something.

Then it would yet again be possible to just add a #clock to the chat buffer and from there on the assistant would always have the up-to-date time.

Therefore,

Suggestion 2

Add a similar live = true (or whatever is the best name) to the variables table.

---

Looking forward to hearing your thoughts on this. Perhaps the time example sounds a bit contrived but I really think this would unlock new capability and increase accuracy and save tokens.

[olimorris]

I'm pretty sure #buffer{watch} does exactly what you're asking for. It sends over the buffer's changes with every request. It's hugely token efficient and what I would define as "live".

When I first saw the pinning feature, I actually thought that it was more inline with the 'live data' concept above. I.e. that it was inserted/appended to the system prompt and that it was replaced/refreshed on each new user message sent. This is for me a more natural meaning of 'pinning'.

Yeah I appreciate this. Coming up with terms for every new feature in CodeCompanion isn't exactly my forte 😆. My thought process for this was that on every "submit", you're sending a new message to your LLM. And you're effectively "pinning" that buffer to every new message rather than pinning the buffer to the chat buffer itself. It was implemented prior to watch and will likely be removed from the plugin soon.

Regarding your ask...can you give me a non-trivial example of what data you'd like to be "live"? I appreciate the time example but I can't foresee that making an iota of difference in quality to any output from an LLM and I have never required anything to be "live" other than buffers.

[olimorris]

Thinking about how this could be implemented, we could introduce the notion of "live variables" into a chat buffer that would be refreshed on every request. Something like The time is ${time} and time would reference a function in a config dictionary. This could be easily enabled in system prompts as well.

[einarpersson]

@olimorris Thank you for considering this. I think it's wise of you to be skeptical in order to avoid feature bloat, but I argue that this is one (1) concept / capability that could unlock a family of creative use cases (instead of addressing them one-by-one). Some may be not obvious right now.

${time} that can be used in system prompts and in chat buffers looks great.

Regarding your ask...can you give me a non-trivial example of what data you'd like to be "live"? I appreciate the time example but I can't foresee that making an iota of difference in quality to any output from an LLM and I have never required anything to be "live" other than buffers.

Sure! I'm gonna think more about this, as I said I have previously used it in a shell context. I'm still getting to know Codecompanion and trying to understand how I want to use it. But here are some initial thoughts/ideas:

Builtins:

• live ${lsp} diagnostics. Always containing the latest diagnostics etc..
• ${viewport}? Perhaps? being able to say 'how about this code?' and knowing it always sees what i see. Kinda cool?
• I would also argue that ${buffer} still makes sense (even though #buffer{watch} does probably takes care of a lot), because sometimes you make many/verbose edits and you just want to have the current version of the draft at hand and not all diffs. When you run format you may change the indentation of all lines for example, causing almost duplication even when doing "watch".

Custom / user live variables:

• ${tests} that contains latest unit test output (how it works is up to the user. Command, filtering etc. depends on the project)
• ${lint} that contains latest linting output
• ${memory} that uses something like mem0 or just naive textfiles to recall memories/context depending on the current conversation.
• ${rag}
• ${git_status} (or ${git_diff} or ...)
• ...

(Note: I also use neovim for a lot of non-coding tasks, eg. note-taking (with obsidian.nvim) and project management. In these cases things like ${time} perhaps make more sense than in a coding context. I know codecompanion IS a code assistant but I will definitely adapt it (eg prompts and tools) to a non-coding environment.)

[olimorris]

I like it. Okay, so thinking of this from a user perspective:

1. I open up a CodeCompanion chat buffer
2. I type Can you explain why my tests are failing? <live>Here is the test output, currently: ${tests}</live> and on submit, CodeCompanion replaces the tests variable with the last output from the test suite (or however we've setup the tests live variable)
3. The LLM responds as it typically would. Probably some thing like It looks like XYZ. Would you like me to fix it?
4. A new user prompt is automatically created and everything in the <live></live> tags is carried forward... <live>Here is the test output, currently: ${tests}</live>|

And the cursor is inserted at |...What do you think?

[einarpersson]

Hi! It sounds like we have two different cases for these live areas (is that what we call them now? :) )

1. Live areas in the original system prompt
2. Live areas that the user is adding along the way

I initially didn't think of having these variables in a <live>-tags but that sounds cool. What do you think of this?

1. Live areas exists in the system prompt and / or the last user message.
2. Variables outside of live areas will be populated once
3. Variables inside live areas will be populated on each submission.
4. Live areas in the last user message will be carried forward to next message as described above.

Also, is it really necessary to introduce new syntax here #foo vs ${foo}? If it is the context (inside or outside a live area) that defines the behaviour, shouldn't the tags be enough? Both #lsp, #viewport and #buffer could be used inside and outside of live area, I think? Are there any variables that are well defined in one but not the other?

[olimorris]

I'm probably a few weeks away from tackling this so will give it all some thought. I think a H3 header called with a specific heading would be even cleaner than the tags.

Regarding syntax, I'll get back to you. There's a potential challenge around how hashtag variables have been implemented and communicated to people and how they'd be implemented.

Ultimately, ease of use has to win out.

[olimorris]

@einarpersson - FYI, I added a prompt decorator feature, yesterday that you may find useful. It decorates a user prompt prior to sending over to an LLM, writing this into the chat history too.

[einarpersson]

Hmm, I suppose what I most of all what is a way to edit the full history / conversation before sending it to the user prompt. Is that possible to do via the API? If I implement a prompt decorator, or if I listen to the CodeCompanionRequestStarted -event (or perhaps CodeCompanionChatSubmitted?), could I at that point intercept and modify the full content (not just the last user message) being sent to the LLM? If so then perhaps I could try to implement (or at least explore) what I am after as an extension, as a POC?

edit: i think i figured it out, gonna experiment some more. I'm really glad that this plugin has been design with extensibility in mind