Can my RAG agent answer common questions?

[Antoine101]

Hi all,

I have been building a little RAG app following one of LlangChain's tutorial. It's based on a single pdf document I load, split, embed, and store in a vector store. The code is provided below.

When asking him questions about something the document talks about, like "Can you give me several statistical learning methods?", it answers pretty correctly.
However, when asking a general question like "What is football?" It answers "I'm sorry, but this document appears to be about statistical learning methods and does not contain a definition of football. Therefore, I cannot answer your question.".

This may be a silly question but how come it is unable to leverage the LLM's internal knowledge to answer questions about stuff that are not covered by the provided document.

Is it due to the LLM used? To the way the graph is created?
How could I modify the provided code to have an agent that's able to construct answers based both on my knowledge base and answer other non related topics?
Is it even something we would like? How would we be sure that it constructs its answers based on the KB or its internal knowledge?

Thanks

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain.chat_models import init_chat_model
from langchain_google_genai import GoogleGenerativeAIEmbeddings
from langchain_core.vectorstores import InMemoryVectorStore
from langgraph.graph import START, MessagesState, StateGraph
from langchain_core.messages import SystemMessage
from langgraph.prebuilt import ToolNode
from langchain_core.tools import tool
from langgraph.graph import END
from langgraph.prebuilt import ToolNode, tools_condition
from langchain_core.documents import Document
from typing import TypedDict, List
from langchain import hub


file_path = r"C:\Users\APU\Documents\Books\An Introduction to Statistical Learning.pdf"

loader = PyPDFLoader(file_path=file_path)
docs = []
docs = loader.load()


text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=2000,
    chunk_overlap=30,
    add_start_index=True)

all_splits = text_splitter.split_documents(docs)

print(f"Split the pdf into {len(all_splits)} sub-documents.")

llm = init_chat_model("gemini-2.0-flash", model_provider="google_genai")
embeddings = GoogleGenerativeAIEmbeddings(model="models/embedding-001")
vector_store = InMemoryVectorStore(embeddings)

document_ids = vector_store.add_documents(documents=all_splits)

prompt = hub.pull('rlm/rag-prompt')

class State(TypedDict):
    question: str
    context: List[Document]
    answer: str

def retrieve(state: State):
    print("--Retrieving Information--")
    question = state['question']
    retrieved_docs = vector_store.similarity_search(query=question,k=4)
    return {'context': retrieved_docs}

def generate(state: State):
    print("--Generating Response--")
    messages = prompt.invoke({'question': state['question'], 'context': state['context']})
    response = llm.invoke(messages)
    return {'answer': response}
        
builder = StateGraph(State)

builder.add_node("retrieve", retrieve)
builder.add_node("generate", generate)

builder.add_edge(START, "retrieve")
builder.add_edge("retrieve", "generate")
builder.add_edge("generate", END)

graph = builder.compile()

for event in graph.stream({'question': 'What is football?'}, stream_mode='values'):
    if event.get('answer',''):
        event['answer'].pretty_print()

[ahouse73]

My guess is:

If you look into the prompt that you load with prompt = hub.pull('rlm/rag-prompt'):

human

You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know >the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.

Question: {question}

Context: {context}

Answer:

You see that its prompted to answer questions based on the context (youre document), which is also what you usally want.
If you change the prompt, it should also answer common logic questions

Hope this helps

[Antoine101]

Thanks for replying so quickly @ahouse73 !

I modified my prompt from the rag-prompt pulled from the hub to the following custom prompt:

template = """Use the following pieces of context to answer the question at the end.
If you don't know the answer, just say that you don't know, don't try to make up an answer.
If the context is not relevant, please answer the question by using your own knowledge about the topic.
Use three sentences maximum and keep the answer as concise as possible.
Always say "thanks for asking!" at the end of the answer.

{context}

Question: {question}

Helpful Answer:"""
prompt = PromptTemplate.from_template(template)

I was expecting the addition of the line "If the context is not relevant, please answer the question by using your own knowledge about the topic." to do the job but I still get:
"I'm sorry, but this document focuses on statistical learning methods and does not contain any information about football. Thus, I cannot answer your question using the context provided. Thanks for asking!"

EDIT:

It works with this other piece of code which is directly taken from LlangChain RAG tuto part 2. Not sure what difference is crucial between the two:

import getpass
import os
from langchain.chat_models import init_chat_model
from langchain_google_genai import GoogleGenerativeAIEmbeddings
from langchain_core.vectorstores import InMemoryVectorStore

import bs4
from langchain import hub
from langchain_community.document_loaders import WebBaseLoader
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langgraph.graph import START, MessagesState, StateGraph
from typing_extensions import List, TypedDict

from langchain_text_splitters import RecursiveCharacterTextSplitter

from langchain import hub
from langchain_core.prompts import PromptTemplate

from langchain_core.documents import Document
from typing_extensions import List, TypedDict

from langchain_core.tools import tool

from langgraph.graph import END
from langgraph.prebuilt import ToolNode, tools_condition

from langchain_core.messages import SystemMessage
from langgraph.prebuilt import ToolNode


if not os.environ.get("GOOGLE_API_KEY"):
  os.environ["GOOGLE_API_KEY"] = getpass.getpass("Enter API key for Google Gemini: ")


llm = init_chat_model("gemini-2.0-flash", model_provider="google_genai")
embeddings = GoogleGenerativeAIEmbeddings(model="models/embedding-001")
vector_store = InMemoryVectorStore(embeddings)

# Load and chunk contents of the blog
loader = WebBaseLoader(
    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
    bs_kwargs=dict(
        parse_only=bs4.SoupStrainer(
            class_=("post-content", "post-title", "post-header")
        )
    ),
)
docs = loader.load()

assert len(docs) == 1
print(f"Total characters: {len(docs[0].page_content)}")

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,  # chunk size (characters)
    chunk_overlap=200,  # chunk overlap (characters)
    add_start_index=True,  # track index in original document
)
all_splits = text_splitter.split_documents(docs)

print(f"Split blog post into {len(all_splits)} sub-documents.")

document_ids = vector_store.add_documents(documents=all_splits)

graph_builder = StateGraph(MessagesState)

@tool(response_format="content_and_artifact")
def retrieve(query: str):
    """Retrieve information related to a query."""
    retrieved_docs = vector_store.similarity_search(query, k=2)
    serialized = "\n\n".join(
        (f"Source: {doc.metadata}\n" f"Content: {doc.page_content}")
        for doc in retrieved_docs
    )
    return serialized, retrieved_docs

# Step 1: Generate an AIMessage that may include a tool-call to be sent.
def query_or_respond(state: MessagesState):
    """Generate tool call for retrieval or respond."""
    llm_with_tools = llm.bind_tools([retrieve])
    response = llm_with_tools.invoke(state["messages"])
    # MessagesState appends messages to state instead of overwriting
    return {"messages": [response]}

# Step 2: Execute the retrieval.
tools = ToolNode([retrieve])

# Step 3: Generate a response using the retrieved content.
def generate(state: MessagesState):
    """Generate answer."""
    # Get generated ToolMessages
    recent_tool_messages = []
    for message in reversed(state["messages"]):
        if message.type == "tool":
            recent_tool_messages.append(message)
        else:
            break
    tool_messages = recent_tool_messages[::-1]

    # Format into prompt
    docs_content = "\n\n".join(doc.content for doc in tool_messages)
    system_message_content = (
        "You are an assistant for question-answering tasks. "
        "Use the following pieces of retrieved context to answer "
        "the question. If the context is not relevant, please answer the question by using your own knowledge about the topic. "
        "Use three sentences maximum and keep the "
        "answer concise."
        "\n\n"
        f"{docs_content}"
    )
    conversation_messages = [
        message
        for message in state["messages"]
        if message.type in ("human", "system")
        or (message.type == "ai" and not message.tool_calls)
    ]
    prompt = [SystemMessage(system_message_content)] + conversation_messages

    # Run
    response = llm.invoke(prompt)
    return {"messages": [response]}

graph_builder.add_node(query_or_respond)
graph_builder.add_node(tools)
graph_builder.add_node(generate)

graph_builder.set_entry_point("query_or_respond")
graph_builder.add_conditional_edges(
    "query_or_respond",
    tools_condition,
    {END: END, "tools": "tools"},
)
graph_builder.add_edge("tools", "generate")
graph_builder.add_edge("generate", END)

graph = graph_builder.compile()

input_message = "Hello"

for step in graph.stream(
    {"messages": [{"role": "user", "content": input_message}]},
    stream_mode="values",
):
    step["messages"][-1].pretty_print()

input_message = "What is football?"

for step in graph.stream(
    {"messages": [{"role": "user", "content": input_message}]},
    stream_mode="values",
):
    step["messages"][-1].pretty_print()

The answer returned to the "What is football?" question is:
"This question is not related to the provided context. Football is a team sport played with a ball, typically involving two teams of 11 players trying to score by advancing the ball into the opposing team's goal. The sport has various forms, with the most popular being association football (soccer) and American football."