Why ConversationalRetrievalChain only answer questions related RAG document and lost original model answering capability

[kunZooberg]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import os
import asyncio

import bs4
from langchain_openai import AzureChatOpenAI
from langchain import hub
from langchain_chroma import Chroma
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.document_loaders import WebBaseLoader
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import AzureOpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

async def run_func():
    model = AzureChatOpenAI(
        openai_api_key= os.environ["AZURE_OPENAI_API_KEY"],
        azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
        azure_deployment=os.environ["AZURE_OPENAI_DEPLOYMENT_NAME"],
        openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
        temperature=0.2,
        streaming=True,
    )

    loader = WebBaseLoader(
        #web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
        web_paths=("https://kubernetes.io/docs/concepts/containers/",),  
        bs_kwargs=dict(
            parse_only=bs4.SoupStrainer(
                class_=("post-content", "post-title", "post-header", "td-content")
            )
        ),
    )
    docs = loader.load()

    text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
    splits = text_splitter.split_documents(docs)
    vectorstore = Chroma.from_documents(documents=splits, embedding=AzureOpenAIEmbeddings(model="text-embedding-3-small"))

    retriever = vectorstore.as_retriever()
    memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
    conversation_chain = ConversationalRetrievalChain.from_llm(
        llm=model,
        retriever=retriever,
        memory=memory
    )

    def rag_chatbot(query):
        response = conversation_chain({"question": query})
        return response['answer']


    print("RAG Chatbot initialized. Type 'exit' to quit.")
    while True:
        user_query = input("You: ")
        if user_query.lower() == 'exit':
            break
        response = rag_chatbot(user_query)
        print(f"Assistant: {response}")


if __name__ == "__main__":
    asyncio.run(run_func())

Description

I am learing RAG + langchain, and write this code, found it only can anwser question related RAG documents. if the anwser is not related, it will print "Assistant: I'm sorry, but I don't have the answer to that question.", but I configured it with chatgpt3.5, what coult be the problem?

System Info

langchain==0.2.14
langchain-anthropic==0.1.23
langchain-chroma==0.1.3
langchain-community==0.2.12
langchain-core==0.2.34
langchain-huggingface==0.0.3
langchain-openai==0.1.22
langchain-text-splitters==0.2.2

platform mac
python 3.12.5

System Information

OS: Darwin
OS Version: Darwin Kernel Version 23.5.0: Wed May 1 20:16:51 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T8103
Python Version: 3.12.5 (main, Aug 6 2024, 19:08:49) [Clang 15.0.0 (clang-1500.3.9.4)]

Package Information

langchain_core: 0.2.34
langchain: 0.2.14
langchain_community: 0.2.12
langsmith: 0.1.101
langchain_anthropic: 0.1.23
langchain_chroma: 0.1.3
langchain_huggingface: 0.0.3
langchain_openai: 0.1.22
langchain_text_splitters: 0.2.2

Optional packages not installed

langgraph
langserve

Other Dependencies

aiohttp: 3.10.5
anthropic: 0.34.1
async-timeout: Installed. No version info available.
chromadb: 0.5.5
dataclasses-json: 0.6.7
defusedxml: 0.7.1
fastapi: 0.112.1
httpx: 0.27.0
huggingface-hub: 0.24.6
jsonpatch: 1.33
numpy: 1.26.4
openai: 1.42.0
orjson: 3.10.7
packaging: 24.1
pydantic: 2.8.2
PyYAML: 6.0.2
requests: 2.32.3
sentence-transformers: 3.0.1
SQLAlchemy: 2.0.32
tenacity: 8.5.0
tiktoken: 0.7.0
tokenizers: 0.19.1
transformers: 4.44.1
typing-extensions: 4.12.2

[dosubot[bot]]

Hello @kunZooberg! I'm here to help you with any bugs, questions, and contributing needs. Let's tackle this issue together!

The ConversationalRetrievalChain is designed to answer questions based on the retrieved documents. This behavior is intentional and is due to the way the chain is configured. The prompt given to the model explicitly instructs it to answer based on the retrieved documents, which limits its responses to the context provided by those documents.

Here's an example of how the prompt is structured:

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are an AI assistant. Answer based on the retrieved documents:"
            "\n<Documents>\n{context}\n</Documents>",
        ),
        ("user", "{question}?"),
    ]
)

This prompt tells the model to base its answers on the retrieved documents, thereby limiting its ability to answer questions outside of the provided context.

Additionally, the ConversationalRetrievalChain handles chat history and new questions by creating a "standalone question" that is passed into the retrieval step to fetch relevant documents. This ensures that the model's responses are always grounded in the retrieved context, further limiting its ability to answer questions outside of the provided documents ^[1][2].

To continue talking to Dosu, mention @dosu.