How should I handle multi-user conversations in a chatbot with the LLM and langchain

[MadDonkey]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from typing import Dict
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import AzureChatOpenAI
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.memory import ChatMessageHistory
from langchain_community.document_loaders import WebBaseLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma
from langchain_openai import AzureOpenAIEmbeddings
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnableBranch



chat_model_api_key = "my api key"
chat_model_deployment = "model_deployment_name"
chat_model_endpoint = "model_endpoint"
chat_model_api_version = "api_version"

embed_model_deployment = "embed_model_deployment_name"
embed_model_endpoint = "embed_model_endpoint"
embed_model_api_key = "embed_model_api_key"
embed_model_api_version = "embed_model_api_version


loader = WebBaseLoader("https://baike.baidu.com/item/%E5%AF%8C%E5%9D%9A%E4%B9%89%E5%8D%9A/188127?fr=ge_ala")
data = loader.load()

text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)
all_splits = text_splitter.split_documents(data)


vectorstore = Chroma.from_documents(
    documents=all_splits,
    embedding=AzureOpenAIEmbeddings(
        azure_deployment=embed_model_deployment,
        api_key=embed_model_api_key,
        azure_endpoint=embed_model_endpoint,
        api_version=embed_model_api_version
    )
)

# k is the number of chunks to retrieve
retriever = vectorstore.as_retriever(k=4)

docs = retriever.invoke("")

chat = AzureChatOpenAI(
    openai_api_version=chat_model_api_version,
    azure_endpoint=chat_model_endpoint,
    azure_deployment=chat_model_deployment,
    api_key=chat_model_api_key,
    temperature=0
)

question_answering_prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "Answer the user's questions based on the below context:\n\n{context}",
        ),
        MessagesPlaceholder(variable_name="messages"),
    ]
)

query_transform_prompt = ChatPromptTemplate.from_messages(
    [
        MessagesPlaceholder(variable_name="messages"),
        (
            "user",
            "Given the above conversation, generate a search query to look up in order to get information relevant to the conversation. Only respond with the query, nothing else.",
        ),
    ]
)

query_transforming_retriever_chain = RunnableBranch(
    (
        lambda x: len(x.get("messages", [])) == 1,
        # If only one message, then we just pass that message's content to retriever
        (lambda x: x["messages"][-1].content) | retriever,
    ),
    # If messages, then we pass inputs to LLM chain to transform the query, then pass to retriever
    query_transform_prompt | chat | StrOutputParser() | retriever,
).with_config(run_name="chat_retriever_chain")

document_chain = create_stuff_documents_chain(chat, question_answering_prompt)


conversational_retrieval_chain = RunnablePassthrough.assign(
    context=query_transforming_retriever_chain,
).assign(
    answer=document_chain,
)

demo_ephemeral_chat_history = ChatMessageHistory()


demo_ephemeral_chat_history.add_user_message("富坚义博老婆是谁?")

response = conversational_retrieval_chain.invoke(
    {"messages": demo_ephemeral_chat_history.messages},
)

demo_ephemeral_chat_history.add_ai_message(response["answer"])

print(response)


demo_ephemeral_chat_history.add_user_message("tell me more about that!")

response = conversational_retrieval_chain.invoke(
    {"messages": demo_ephemeral_chat_history.messages}
)

print(response)

Description

I'm trying to use the langchain and LLM to develop a chatbot. The demo code is above, now I want to implement a chatbot that can support multi-user conversations. If there are multiple sessions initiated at the same time, do I need to maintain the context of each user in the code to ensure that the robot can answer the question correctly? For example, if two users ask a question at the same time, user A asks what 1+1 is, and user B asks how many people live in the world, how can the chatbot guarantee that user B's answer will not be returned to user A? Do I need to maintain the user context on my own or langchain or the big model can do it on its own？

System Info

langchain==0.1.9
langchain-chroma==0.1.1
langchain-community==0.0.37
langchain-core==0.1.52
langchain-openai==0.1.6
langchain-text-splitters==0.0.2

paltform:mac
Python:3.9.6

[dosubot[bot]]

Hey @MadDonkey! I'm Dosu, your friendly bot here to help you squash bugs, answer questions, and even guide you to become a contributor. I'm here to assist you while we wait for a human maintainer. Taking a look at your issue on multi-user conversations now and will get back to you shortly with a full answer! 🚀

[maximeperrindev]

Hi @MadDonkey,

https://python.langchain.com/v0.1/docs/expression_language/how_to/message_history/

[dosubot[bot]]

To implement a multi-user chatbot using LangChain and an LLM, you need to manage the context for each user independently. LangChain does not automatically handle user context separation, so you will need to implement this in your code.

Here's how you can modify your existing code to support multi-user conversations by maintaining separate chat histories for each user:

1. Create a dictionary to store chat histories for each user.
2. Modify the conversational_retrieval_chain to handle user-specific contexts.
3. Ensure that each user's messages are processed independently.

Below is the modified version of your code to support multi-user conversations:

from typing import Dict
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import AzureChatOpenAI
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.memory import ChatMessageHistory
from langchain_community.document_loaders import WebBaseLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma
from langchain_openai import AzureOpenAIEmbeddings
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnableBranch

chat_model_api_key = "my api key"
chat_model_deployment = "model_deployment_name"
chat_model_endpoint = "model_endpoint"
chat_model_api_version = "api_version"

embed_model_deployment = "embed_model_deployment_name"
embed_model_endpoint = "embed_model_endpoint"
embed_model_api_key = "embed_model_api_key"
embed_model_api_version = "embed_model_api_version"

loader = WebBaseLoader("https://baike.baidu.com/item/%E5%AF%8C%E5%9D%9A%E4%B9%89%E5%8D%9A/188127?fr=ge_ala")
data = loader.load()

text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)
all_splits = text_splitter.split_documents(data)

vectorstore = Chroma.from_documents(
    documents=all_splits,
    embedding=AzureOpenAIEmbeddings(
        azure_deployment=embed_model_deployment,
        api_key=embed_model_api_key,
        azure_endpoint=embed_model_endpoint,
        api_version=embed_model_api_version
    )
)

# k is the number of chunks to retrieve
retriever = vectorstore.as_retriever(k=4)

docs = retriever.invoke("")

chat = AzureChatOpenAI(
    openai_api_version=chat_model_api_version,
    azure_endpoint=chat_model_endpoint,
    azure_deployment=chat_model_deployment,
    api_key=chat_model_api_key,
    temperature=0
)

question_answering_prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "Answer the user's questions based on the below context:\n\n{context}",
        ),
        MessagesPlaceholder(variable_name="messages"),
    ]
)

query_transform_prompt = ChatPromptTemplate.from_messages(
    [
        MessagesPlaceholder(variable_name="messages"),
        (
            "user",
            "Given the above conversation, generate a search query to look up in order to get information relevant to the conversation. Only respond with the query, nothing else.",
        ),
    ]
)

query_transforming_retriever_chain = RunnableBranch(
    (
        lambda x: len(x.get("messages", [])) == 1,
        # If only one message, then we just pass that message's content to retriever
        (lambda x: x["messages"][-1].content) | retriever,
    ),
    # If messages, then we pass inputs to LLM chain to transform the query, then pass to retriever
    query_transform_prompt | chat | StrOutputParser() | retriever,
).with_config(run_name="chat_retriever_chain")

document_chain = create_stuff_documents_chain(chat, question_answering_prompt)

conversational_retrieval_chain = RunnablePassthrough.assign(
    context=query_transforming_retriever_chain,
).assign(
    answer=document_chain,
)

# Dictionary to store chat histories for each user
user_chat_histories: Dict[str, ChatMessageHistory] = {}

def get_user_chat_history(user_id: str) -> ChatMessageHistory:
    if user_id not in user_chat_histories:
        user_chat_histories[user_id] = ChatMessageHistory()
    return user_chat_histories[user_id]

def handle_user_message(user_id: str, message: str) -> str:
    chat_history = get_user_chat_history(user_id)
    chat_history.add_user_message(message)
    
    response = conversational_retrieval_chain.invoke(
        {"messages": chat_history.messages}
    )
    
    chat_history.add_ai_message(response["answer"])
    return response["answer"]

# Example usage
user_id_1 = "user_1"
user_id_2 = "user_2"

message_1 = "What is 1+1?"
message_2 = "How many people live in the world?"

response_1 = handle_user_message(user_id_1, message_1)
response_2 = handle_user_message(user_id_2, message_2)

print(f"Response to user 1: {response_1}")
print(f"Response to user 2: {response_2}")

Explanation:

1. User Chat Histories: A dictionary user_chat_histories is used to store ChatMessageHistory objects for each user.
2. get_user_chat_history: This function retrieves the chat history for a given user, creating a new one if it doesn't exist.
3. handle_user_message: This function processes a user's message by adding it to their chat history, invoking the conversational_retrieval_chain, and then adding the AI's response to the chat history.

By maintaining separate chat histories for each user, you ensure that the context for each user is preserved independently, preventing any mix-up of responses between different users.

Details

Sources

• How to implement system prompt and better chat history.
• Need some suggestion on using langchain for doc search
• Plugin chat gpt
• libs/langchain/langchain/schema/chat.py
• docs/docs/how_to/qa_per_user.ipynb
• libs/experimental/langchain_experimental/chat_models/llm_wrapper.py
• libs/core/langchain_core/chat_history.py
• libs/community/langchain_community/chat_message_histories/file.py
• libs/langchain/langchain/smith/evaluation/runner_utils.py
• libs/core/langchain_core/language_models/llms.py
• libs/core/langchain_core/caches.py
• libs/core/langchain_core/language_models/chat_models.py
• libs/community/langchain_community/indexes/_document_manager.py
• docs/docs/integrations/vectorstores/redis.ipynb
• libs/community/langchain_community/storage/redis.py
• libs/community/langchain_community/storage/mongodb.py
• docs/docs/integrations/providers/redis.mdx

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[evertonaleixo]

@dosu, does not make sense to use chat history from all users and add in system prompt a mention to generate a response to the user of ID 'user_1'? Because the user_2 can also interactiing with user_1, and his messages could be important as part of context.