Inject ChatMemory into the Prompt

[andreadimaio]

Hi, I'd like to share an idea that could speed-up the process of creating a prompt that leverage ChatMemory for AI interactions.
The focus of this proposal is to simplify how developers pass conversation history into prompts.

Problem

Currently, when developing an AI service that utilizes a conversation history as part of the prompt, the developer is responsible for manually formatting the chat history. This involves extracting and formatting past user and assistant messages into a structured format.

Consider the example below, where the developer needs to inject a conversation history (as a formatted string) into the prompt:

@RegisterAiService(
   chatMemoryProviderSupplier = NoChatMemoryProviderSupplier.class 
)
public interface AIService {

    @SystemMessage("""
        Given the following conversation and a follow-up question, rephrase the follow-up question to be a
        standalone question, in its original language. Return the follow-up question VERBATIM if the
        question is changing the topic with respect to the conversation. It is **VERY IMPORTANT** to only
        output the rephrased standalone question; do not add notes or comments in the output.

        Chat History:
        {chatHistory}""")
    public String rephrase(String chatHistory, @UserMessage String question);
}

Where the chatHistory placeholder has the format:

User: <message> 
Assistant: <message> ...

To implement this example, the developer needs to inject the ChatMemoryStore in a service class and generate the correct format for the chatHistory placeholder.

Simplifying ChatMemory Injection

To accelerate this process and reduce manual work, it could be useful to have a new feature that allows the ChatMemoryStore/ChatMemory to be passed directly as a method parameter. In this case a specific implementation will be used to automatically map the ChatMemoryStore/ChatMemory to the appropriate format that the developer requires for the prompt, where the default format could be what I wrote above.

@RegisterAiService(
   chatMemoryProviderSupplier = NoChatMemoryProviderSupplier.class 
)
public interface AIService {

    @SystemMessage("""
        Given the following conversation and a follow-up question, rephrase the follow-up question to be a
        standalone question, in its original language. Return the follow-up question VERBATIM if the
        question is changing the topic with respect to the conversation. It is **VERY IMPORTANT** to only
        output the rephrased standalone question; do not add notes or comments in the output.

        Chat History:
        {chatHistory}""")
    public String rephrase(ChatMemoryStore chatHistory,  @UserMessage String question);
}

Customizable Format:

If the developer requires a different conversation format, they could customize the formatting by adding a property in the @RegisterAiService annotation. For example:

@RegisterAiService(
   chatMemoryProviderSupplier = NoChatMemoryProviderSupplier.class ,
   chatFormat = CustomChatFormat.class
)
public interface AIService {
    ...
}

This is an idea that came to mind while using quarkus-langchain4j. I don't know if it makes sense or if there is already something that can be used to do this operation, let me know 😄.

[geoand]

This is indeed interesting and I think it's somewhat related to @io.quarkiverse.langchain4j.SeedMemory. WDYT?

[andreadimaio]

It could be. In my opinion, @io.quarkiverse.langchain4j.SeedMemory is something that can be used to show the LLM how to behave and avoid unexpected results.

[andreadimaio]

What do you think about the "usability"? Could it be a good way to allow the developer to customise using the @RegisterAiService? If this is ok for you, this could be moved as an issue.

[geoand]

Yeah, that's perfectly fine

[andreadimaio]

I'm thinking about how to control the use of the custom format and whether it's better to have this as a property of the @RegisterAiService annotation or to create a new annotation that can be applied at class or method level to give the developer more control. From my point of view, the second way might be better.

[geoand]

I am generally weary of adding new annotations since they are not as easily discoverable by users