Client side chat memory?

[patriot1burke]

Chat memory greatly bothers me. Holding session state in server memory does not scale well and you have to have a clustered cache and/or sticky sessions with the client.

I thought: Why not serialize chat memory and send it back to the client and have the client resend it with each chat message?

Sound good? Here's an initial implementation:

• The Implementation was pretty simple. It's just a @RequestScoped bean that is injected into the application and filled by the application, and then serialized when the user message has been processed.
• The CDI Integration was a little hacky because Quarkus tries to evict chat memory when beans go out of scope.

Interested in getting a PR for this once I flush out the details?

I've also have had HUGE problems with getting the AI to format complex responses correctly, so instead I've been piggy-backing data with the response. Eventually, I'm going to have a client Response Context object that contains the current ChatMemory and a set of UI events that is calculated by a tool and attached the the context returned to the client. The client will then output the chat response from the AI and loop through any piggy-backed UI events to render relevant information.

[geoand]

👋🏽

I thought: Why not serialize chat memory and send it back to the client and have the client resend it with each chat message?

I must be missing something, but isn't this what the client currently does?

[cescoffier]

It would only work for simple cases:

• no rag (they augment the prompt)
• no guardrails (they can manipulate the memory)
• no memory compression
• no Agentic (as soon as you have multiple ai services involved in a single turn, with different memory)

I think the 3 first cases could work. It will just be slightly different.

One thing to keep in mind is that the framework need to handle the LLM context window. So here it will be the responsibility of the browser, meaning we need to pass the LLM responses to the browser (which contain the token utilization).

[patriot1burke]

It would only work for simple cases:

Because?

• no rag (they augment the prompt)
• no guardrails (they can manipulate the memory)
• no memory compression
• no Agentic (as soon as you have multiple ai services involved in a single turn, with different memory)

I think the 3 first cases could work. It will just be slightly different.

Not understanding why this wouldn't work for any of the cases you listed. This is no different than having a sql ChatMemoryStore.  The client is just holding and propagating the persistent state of chat memory. No reason the state couldn't be opaque to the client hell, even encrypted. Granted, my implementation may need to expand to allow for multiple chat memories.

One thing to keep in mind is that the framework need to handle the LLM context window. So here it will be the responsibility of the browser, meaning we need to pass the LLM responses to the browser (which contain the token utilization).

Not understanding what you're saying here. It's just ChatMemory which is just a list of messages which can be serialized to json.

[cescoffier]

The main difference is that you think there is just one chat memory. We are seeing an increasing number of cases where multiple components interact with (potentially different) LLMs, resulting in several memories - and this for a single "request"

[patriot1burke]

Sure, but multiple chat memories is something the app developer has to manage and control and configure and juggle even without this approach. ChatMemory for langchain4j is not very sophisticated.

Even with multiple chat memories for multiple LLM calls, the whole interaction with various LLMs is still part of a logical user "session" and thus no reason it can't be stored on the client.

I think it makes more sense to implement this as a ChatMemoryStore. Before I thought langchain4j was making some assumptions and caching chat memory, but in looking at Message/TokenWindowChatMemory implementation, it looks like they are delegating to ChatMemoryStore for everything.

[cescoffier]

Yeah, at the moment it’s simple. It starts to be more complicated in agentic architectures. That being said, how we designed it is meant to be serializable and thus could be sent back to the client and restored on every call.