How to use placeholders for variables in my system prompt without "raise KeyError"

[humphreyde]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import bs4
from langchain.chains import create_history_aware_retriever, create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_chroma import Chroma
from langchain_community.chat_message_histories import ChatMessageHistory
from langchain_community.document_loaders import WebBaseLoader
from langchain_core.chat_history import BaseChatMessageHistory
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)


### Construct retriever ###
loader = WebBaseLoader(
    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
    bs_kwargs=dict(
        parse_only=bs4.SoupStrainer(
            class_=("post-content", "post-title", "post-header")
        )
    ),
)
docs = loader.load()

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
splits = text_splitter.split_documents(docs)
vectorstore = Chroma.from_documents(documents=splits, embedding=OpenAIEmbeddings())
retriever = vectorstore.as_retriever()


### Contextualize question ###
contextualize_q_system_prompt = (
    "Given a chat history and the latest user question "
    "which might reference context in the chat history, "
    "formulate a standalone question which can be understood "
    "without the chat history. Do NOT answer the question, "
    "just reformulate it if needed and otherwise return it as is."
)
contextualize_q_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", contextualize_q_system_prompt),
        MessagesPlaceholder("chat_history"),
        ("human", "{input}"),
    ]
)
history_aware_retriever = create_history_aware_retriever(
    llm, retriever, contextualize_q_prompt
)


### Answer question ###
system_prompt = (
    "examples: {example}"
    "\n\n"
    "{context}"
)
qa_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system_prompt),
        MessagesPlaceholder("chat_history"),
        ("human", "{input}"),
    ]
)
question_answer_chain = create_stuff_documents_chain(llm, qa_prompt)

rag_chain = create_retrieval_chain(history_aware_retriever, question_answer_chain)


### Statefully manage chat history ###
store = {}


def get_session_history(session_id: str) -> BaseChatMessageHistory:
    if session_id not in store:
        store[session_id] = ChatMessageHistory()
    return store[session_id]


conversational_rag_chain = RunnableWithMessageHistory(
    rag_chain,
    get_session_history,
    input_messages_key="input",
    history_messages_key="chat_history",
    output_messages_key="answer",
)

Description

{example} is just a example.
As you can see,if there are many "{x},{xx}..." in system prompt, KeyError will be raised, how to handle this problem?
Thanks!
ps: I want to use RunnableWithMessageHistory rather than FewShotPromptTemplate.

System Info

System Information

OS: Linux
OS Version: #1 SMP PREEMPT Mon May 6 09:56:39 PDT 2024
Python Version: 3.10.14 (main, May 6 2024, 19:36:58) [GCC 11.2.0]

Package Information

langchain_core: 0.2.33
langchain: 0.2.14
langchain_community: 0.2.12
langsmith: 0.1.99
langchain_huggingface: 0.0.3
langchain_openai: 0.1.22
langchain_text_splitters: 0.2.2

Optional packages not installed

langgraph
langserve

Other Dependencies

aiohttp: 3.10.5
async-timeout: 4.0.3
dataclasses-json: 0.6.7
huggingface-hub: 0.24.6
jsonpatch: 1.33
numpy: 1.26.4
openai: 1.41.1
orjson: 3.10.7
packaging: 24.1
pydantic: 1.10.17
PyYAML: 6.0.2
requests: 2.32.3
sentence-transformers: 3.0.1
SQLAlchemy: 2.0.32
tenacity: 8.5.0
tiktoken: 0.7.0
tokenizers: 0.19.1
transformers: 4.44.0
typing-extensions: 4.12.2

[dosubot[bot]]

Hey there, @humphreyde! I'm here to help you with your issue. Let's tackle this together!

To use placeholders for variables in your system prompt without raising a KeyError when using RunnableWithMessageHistory, you can make the placeholder optional. This way, if the variable is not provided, it will not raise an error but will return an empty list instead.

Here is an example of how to use the MessagesPlaceholder with the optional parameter set to True:

from langchain_core.prompts import MessagesPlaceholder

# Create a MessagesPlaceholder with the variable name "chat_history" and make it optional
prompt = MessagesPlaceholder("chat_history", optional=True)

# Format messages without providing the "chat_history" variable
formatted_messages = prompt.format_messages()
# This will return an empty list instead of raising a KeyError
print(formatted_messages)  # Output: []

# Format messages with the "chat_history" variable provided
formatted_messages = prompt.format_messages(
    chat_history=[
        ("system", "You are an AI assistant."),
        ("human", "Hello!"),
    ]
)
# This will return the formatted messages
print(formatted_messages)
# Output: [
#     SystemMessage(content="You are an AI assistant."),
#     HumanMessage(content="Hello!"),
# ]

By setting optional=True, you ensure that the placeholder can be used without raising a KeyError if the variable is not provided.

Additionally, you can use partial variables to pre-fill some of the placeholders, as shown in the following example:

from langchain_core.prompts import SystemMessagePromptTemplate

graph_creator_content = """
Your instructions are:
{instructions}
History:
{chat_history}
"""
json_prompt_instructions = {}
graph_analyst_template = SystemMessagePromptTemplate.from_template(
    template=graph_creator_content,
    input_variables=["chat_history"],
    partial_variables={"instructions": json_prompt_instructions},
)
formatted_message = graph_analyst_template.format(chat_history="history")
print(formatted_message)
# Output: SystemMessage(content="\n    Your instructions are:\n    {}\n    History:\n    history\n    ")

This approach allows you to handle placeholders more flexibly and avoid errors when some variables are not provided.

To integrate this with RunnableWithMessageHistory, you can set up your chain and prompt template as follows:

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain_core.messages import HumanMessage, AIMessage, SystemMessage
from langchain_openai import ChatOpenAI

# Define the prompt template with a MessagesPlaceholder
system_prompt = (
    "examples: {example}"
    "\n\n"
    "{context}"
)
qa_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system_prompt),
        MessagesPlaceholder("chat_history", optional=True),
        ("human", "{input}"),
    ]
)

# Define the model and chain
llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)
chain = qa_prompt | llm

# Define the function to get session history
store = {}
def get_session_history(session_id: str):
    if session_id not in store:
        store[session_id] = ChatMessageHistory()
    return store[session_id]

# Wrap the chain with RunnableWithMessageHistory
conversational_rag_chain = RunnableWithMessageHistory(
    chain,
    get_session_history,
    input_messages_key="input",
    history_messages_key="chat_history",
    output_messages_key="answer",
)

# Define the config with session_id
config = {"configurable": {"session_id": "abc5"}}

# Invoke the chain with the message history
response = conversational_rag_chain.invoke(
    {"input": "Hi! I'm Jim"},
    config=config,
)

print(response.content)

By following this approach, you can effectively manage placeholders and avoid KeyError while using RunnableWithMessageHistory ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[humphreyde]

@dosu
for this situation, will raise
raise KeyError( KeyError: "Input to ChatPromptTemplate is missing variables {'test'}

please use partial variables, the original code are:
`import bs4
from langchain.chains import create_history_aware_retriever, create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_chroma import Chroma
from langchain_community.chat_message_histories import ChatMessageHistory
from langchain_community.document_loaders import WebBaseLoader
from langchain_core.chat_history import BaseChatMessageHistory
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)

Construct retriever

loader = WebBaseLoader(
web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
bs_kwargs=dict(
parse_only=bs4.SoupStrainer(
class_=("post-content", "post-title", "post-header")
)
),
)
docs = loader.load()

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
splits = text_splitter.split_documents(docs)
vectorstore = Chroma.from_documents(documents=splits, embedding=OpenAIEmbeddings())
retriever = vectorstore.as_retriever()

Contextualize question

contextualize_q_system_prompt = (
"Given a chat history and the latest user question "
"which might reference context in the chat history, "
"formulate a standalone question which can be understood "
"without the chat history. Do NOT answer the question, "
"just reformulate it if needed and otherwise return it as is."
)
contextualize_q_prompt = ChatPromptTemplate.from_messages(
[
("system", contextualize_q_system_prompt),
MessagesPlaceholder("chat_history"),
("human", "{input}"),
]
)
history_aware_retriever = create_history_aware_retriever(
llm, retriever, contextualize_q_prompt
)

Answer question

system_prompt = (
"{x},{xx},{xxxx},{xxx},{xx},{xxxxxxx},{xxxxxxxx}..."
"\n\n"
"{context}"
)
qa_prompt = ChatPromptTemplate.from_messages(
[
("system", system_prompt),
MessagesPlaceholder("chat_history"),
("human", "{input}"),
]
)
question_answer_chain = create_stuff_documents_chain(llm, qa_prompt)

rag_chain = create_retrieval_chain(history_aware_retriever, question_answer_chain)

Statefully manage chat history

store = {}

def get_session_history(session_id: str) -> BaseChatMessageHistory:
if session_id not in store:
store[session_id] = ChatMessageHistory()
return store[session_id]

conversational_rag_chain = RunnableWithMessageHistory(
rag_chain,
get_session_history,
input_messages_key="input",
history_messages_key="chat_history",
output_messages_key="answer",
)`

[dosubot[bot]]

To avoid a KeyError while keeping placeholders like {x},{xx},{xxxx},{xxx},{xx},{xxxxxxx},{xxxxxxxx} in your ChatPromptTemplate, you can use the partial method to provide default values for these placeholders. This ensures that if these placeholders are not provided during formatting, the default values will be used.

Here's how you can modify your code to use partial variables:

import bs4
from langchain.chains import create_history_aware_retriever, create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_chroma import Chroma
from langchain_community.chat_message_histories import ChatMessageHistory
from langchain_community.document_loaders import WebBaseLoader
from langchain_core.chat_history import BaseChatMessageHistory
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)

# Construct retriever
loader = WebBaseLoader(
    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
    bs_kwargs=dict(
        parse_only=bs4.SoupStrainer(
            class_=("post-content", "post-title", "post-header")
        )
    ),
)
docs = loader.load()

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
splits = text_splitter.split_documents(docs)
vectorstore = Chroma.from_documents(documents=splits, embedding=OpenAIEmbeddings())
retriever = vectorstore.as_retriever()

# Contextualize question
contextualize_q_system_prompt = (
    "Given a chat history and the latest user question "
    "which might reference context in the chat history, "
    "formulate a standalone question which can be understood "
    "without the chat history. Do NOT answer the question, "
    "just reformulate it if needed and otherwise return it as is."
)
contextualize_q_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", contextualize_q_system_prompt),
        MessagesPlaceholder("chat_history"),
        ("human", "{input}"),
    ]
)
history_aware_retriever = create_history_aware_retriever(
    llm, retriever, contextualize_q_prompt
)

# Answer question
system_prompt = (
    "{x},{xx},{xxxx},{xxx},{xx},{xxxxxxx},{xxxxxxxx}..."
    "\n\n"
    "{context}"
)
qa_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system_prompt),
        MessagesPlaceholder("chat_history"),
        ("human", "{input}"),
    ]
)

# Use partial method to provide default values for placeholders
qa_prompt_partial = qa_prompt.partial(
    x="default_x",
    xx="default_xx",
    xxxx="default_xxxx",
    xxx="default_xxx",
    xxxxxxx="default_xxxxxxx",
    xxxxxxxx="default_xxxxxxxx"
)

question_answer_chain = create_stuff_documents_chain(llm, qa_prompt_partial)
rag_chain = create_retrieval_chain(history_aware_retriever, question_answer_chain)

# Statefully manage chat history
store = {}

def get_session_history(session_id: str) -> BaseChatMessageHistory:
    if session_id not in store:
        store[session_id] = ChatMessageHistory()
    return store[session_id]

conversational_rag_chain = RunnableWithMessageHistory(
    rag_chain,
    get_session_history,
    input_messages_key="input",
    history_messages_key="chat_history",
    output_messages_key="answer",
)

# Example invocation
config = {"configurable": {"session_id": "abc5"}}
response = conversational_rag_chain.invoke(
    {"input": "Hi! I'm Jim"},
    config=config,
)

print(response.content)

In this example, the partial method is used to provide default values for the placeholders {x}, {xx}, {xxxx}, {xxx}, {xxxxxxx}, and {xxxxxxxx}. This ensures that if these placeholders are not provided during formatting, the default values will be used, thus avoiding a KeyError ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[humphreyde]

@dosu
when prompts are:
# Answer question system_prompt = ( "{x},{xx},{xxxx},{xxx},{xx},{xxxxxxx},{xxxxxxxx}..." "\n\n" "{context}" )
what I want likes below:

qa_prompt_partial = qa_prompt.partial(
    x="{x}",
    xx="{xx}",
    xxxx="{xxxx}",
    xxx="{xxx}",
    xxxxxxx="{xxxxxxx}",
    xxxxxxxx="{xxxxxxxx}"
)

not:

qa_prompt_partial = qa_prompt.partial(
    x="default_x",
    xx="default_xx",
    xxxx="default_xxxx",
    xxx="default_xxx",
    xxxxxxx="default_xxxxxxx",
    xxxxxxxx="default_xxxxxxxx"
)

[humphreyde]

@dosu
when prompts are:
# Answer question system_prompt = ( "{x},{xx},{xxxx},{xxx},{xx},{xxxxxxx},{xxxxxxxx}..." "\n\n" "{context}" )
what I want likes below:

qa_prompt_partial = qa_prompt.partial(
    x="{x}",
    xx="{xx}",
    xxxx="{xxxx}",
    xxx="{xxx}",
    xxxxxxx="{xxxxxxx}",
    xxxxxxxx="{xxxxxxxx}"
)

not:

qa_prompt_partial = qa_prompt.partial(
    x="default_x",
    xx="default_xx",
    xxxx="default_xxxx",
    xxx="default_xxx",
    xxxxxxx="default_xxxxxxx",
    xxxxxxxx="default_xxxxxxxx"
)

[dosubot[bot]]

To keep the placeholders like {x}, {xx}, {xxxx}, {xxx}, {xxxxxxx}, and {xxxxxxxx} in your ChatPromptTemplate without replacing them with default values, you can use the partial method without specifying values for those placeholders. This way, the placeholders will remain in the template.

Here's how you can achieve this:

import bs4
from langchain.chains import create_history_aware_retriever, create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_chroma import Chroma
from langchain_community.chat_message_histories import ChatMessageHistory
from langchain_community.document_loaders import WebBaseLoader
from langchain_core.chat_history import BaseChatMessageHistory
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)

# Construct retriever
loader = WebBaseLoader(
    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
    bs_kwargs=dict(
        parse_only=bs4.SoupStrainer(
            class_=("post-content", "post-title", "post-header")
        )
    ),
)
docs = loader.load()

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
splits = text_splitter.split_documents(docs)
vectorstore = Chroma.from_documents(documents=splits, embedding=OpenAIEmbeddings())
retriever = vectorstore.as_retriever()

# Contextualize question
contextualize_q_system_prompt = (
    "Given a chat history and the latest user question "
    "which might reference context in the chat history, "
    "formulate a standalone question which can be understood "
    "without the chat history. Do NOT answer the question, "
    "just reformulate it if needed and otherwise return it as is."
)
contextualize_q_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", contextualize_q_system_prompt),
        MessagesPlaceholder("chat_history"),
        ("human", "{input}"),
    ]
)
history_aware_retriever = create_history_aware_retriever(
    llm, retriever, contextualize_q_prompt
)

# Answer question
system_prompt = (
    "{x},{xx},{xxxx},{xxx},{xx},{xxxxxxx},{xxxxxxxx}..."
    "\n\n"
    "{context}"
)
qa_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system_prompt),
        MessagesPlaceholder("chat_history"),
        ("human", "{input}"),
    ]
)

# Use partial method without specifying values for the placeholders you want to keep
qa_prompt_partial = qa_prompt.partial()

question_answer_chain = create_stuff_documents_chain(llm, qa_prompt_partial)
rag_chain = create_retrieval_chain(history_aware_retriever, question_answer_chain)

# Statefully manage chat history
store = {}

def get_session_history(session_id: str) -> BaseChatMessageHistory:
    if session_id not in store:
        store[session_id] = ChatMessageHistory()
    return store[session_id]

conversational_rag_chain = RunnableWithMessageHistory(
    rag_chain,
    get_session_history,
    input_messages_key="input",
    history_messages_key="chat_history",
    output_messages_key="answer",
)

# Example invocation
config = {"configurable": {"session_id": "abc5"}}
response = conversational_rag_chain.invoke(
    {"input": "Hi! I'm Jim"},
    config=config,
)

print(response.content)

In this example, the partial method is used without specifying values for the placeholders {x}, {xx}, {xxxx}, {xxx}, {xxxxxxx}, and {xxxxxxxx}. This ensures that these placeholders will remain in the template and will not be replaced with any default values ^[1][2].

To continue talking to Dosu, mention @dosu.