How do i add memory to RetrievalQA.from_chain_type? or, how do I add a custom prompt to ConversationalRetrievalChain?

[hudsonetkin]

How do i add memory to RetrievalQA.from_chain_type? or, how do I add a custom prompt to ConversationalRetrievalChain?

For the past 2 weeks ive been trying to make a chatbot that can chat over documents (so not in just a semantic search/qa so with memory) but also with a custom prompt. I've tried every combination of all the chains and so far the closest I've gotten is ConversationalRetrievalChain, but without custom prompts, and RetrievalQA.from_chain_type but without memory

[Les-El]

This sounds like maybe the same issue I'm having?

My agent has a tool that references a vectorstore with a codebase. But the agent sends a modified version of the question to the tool (I'm using ConversationSummaryBufferMemory for the agent and ReadOnlySharedMemory for the tool.) But the messages seems to get garbled on the way to and/or from the tool, which seems to lead to hallucinations.

I'd be very interested to know if this has been your experience, and if you've found a solution yet. Cheers.

[DennisMinn]

Encountering the exact same issue, I was thinking about creating a custom chain that modifies ConversationalRetrievalChain source code to answer new_inputs["question"] or a SequentialChain that uses ConversationChain to reformat the question and then passes the result to RetrievalQA. If you have an update to the issue, please let us know.

[DennisMinn]

The solution is actually a lot simpler...

I combined the examples provided by the ConversationalRetrievalChain and StuffDocumentsChain but modify prompt for StuffDocumentsChain to langchain.chains.question_answering.stuff_prompt.PROMPT_SELECTOR.get_prompt(llm)

Modified StuffDocumentsChain

from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate
from langchain.llms import OpenAI

from langchain.chains.question_answering.stuff_prompt import PROMPT_SELECTOR
from langchain.chains.combine_documents.stuff import StuffDocumentsChain

# This controls how each document will be formatted. Specifically,
# it will be passed to `format_document` - see that function for more
# details.
document_prompt = PromptTemplate(
    input_variables=["page_content"],
     template="{page_content}"
)
document_variable_name = "context"
llm = OpenAI()
# The prompt here should take as an input variable the
# `document_variable_name`
prompt = PROMPT_SELECTOR.get_prompt(llm) # This is the modified line

llm_chain = LLMChain(llm=llm, prompt=prompt)
combine_docs_chain = StuffDocumentsChain(
    llm_chain=llm_chain,
    document_prompt=document_prompt,
    document_variable_name=document_variable_name
)

ConversationalRetrievalChain Example

from langchain.chains import (
    StuffDocumentsChain, LLMChain, ConversationalRetrievalChain
)
from langchain.prompts import PromptTemplate
from langchain.llms import OpenAI

combine_docs_chain = StuffDocumentsChain(...)
vectorstore = ...
retriever = vectorstore.as_retriever()

# This controls how the standalone question is generated.
# Should take `chat_history` and `question` as input variables.
template = (
    "Combine the chat history and follow up question into "
    "a standalone question. Chat History: {chat_history}"
    "Follow up question: {question}"
)
prompt = PromptTemplate.from_template(template)
llm = OpenAI()
question_generator_chain = LLMChain(llm=llm, prompt=prompt)
chain = ConversationalRetrievalChain(
    combine_docs_chain=combine_docs_chain,
    retriever=retriever,
    question_generator=question_generator_chain,
)

[saichandra1199]

template = """ You are a knowledgeable chatbot, here to help with questions of the user. Your tone should be professional and informative.
    
    Context: {context}
    History: {history}

    User: {question}
    Chatbot:
    """
prompt = PromptTemplate(
        input_variables=["history", "context", "question"],
        template=template,
    )

memory = ConversationBufferMemory(
        memory_key="history",
        return_messages=True,
        input_key="question"
    )
    
  qa_chain = RetrievalQA.from_chain_type(
            llm=llm,
            chain_type='stuff',
            retriever=retriever,
            verbose=True,
            chain_type_kwargs={
                "verbose": True,
                "prompt": prompt,
                "memory": memory,
            }
        )
        
qa_chain.invoke({"query": query})