Getting the documents used by an Agent

[eduardoscamargo]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_community.document_loaders import WebBaseLoader
from langchain_community.vectorstores import FAISS
from langchain_openai import AzureOpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain.tools.retriever import create_retriever_tool
from langchain_core.messages import SystemMessage, HumanMessage

loader = WebBaseLoader("https://www.example1.com/gifts", "https://www.example2.com/best-gifts")
docs = loader.load()
documents = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200).split_documents(docs)
vectordb = FAISS.from_documents(documents, AzureOpenAIEmbeddings())
retriever = vectordb.as_retriever()
retriever

gift_retriever_tool = create_retriever_tool(
    retriever,
    "gift_search",
    "Use this tool if the user is looking for something to purchase or needs help finding a product to buy"
)

tools = [gift_retriever_tool]
model = chat.bind_tools(tools) # Chat is an Azure OpenAI model

gift_retriever_tool.invoke(model.invoke([
  SystemMessage(content="You are a helpful assistant that helps the user in their search. Provide at maximum 3 items/suggestions."),
  HumanMessage(content="Gifts for a boy")
]).tool_calls[0])

Description

The code above works flawlessly, when I invoke the agent that uses the model above, it "reads" the two example webpages, creates documents for them, and searches them. However, I need to know which documents were used by the agent so I can know what URL(s) was(were) used. If I run something like vectordb.similarity_search('gifts for boys') I get the documents with the metadata I want:

Document(metadata={'source': 'https://www.example1.com/gifts', 'title': '22 Best Gifts for Teens, According to Experts (and Teens).', 'description': 'Find the best Christmas...

However, if I run the agent, I don't have access to documents, I have only the final answer from the agent.
The code above is simplified, I am using LangGraph, but the question is the same, how do I access the document metadata in the "middle" of the processing? I tried using callbacks, but I had no luck and it doesn't seem to be the right solution.

I spent days on this, reading everything I could, but I couldn't figure out how to do it.

System Info

Langchain version:

langchain==0.2.16
langchain-community==0.2.16
langchain-core==0.2.38
langchain-elasticsearch==0.2.2
langchain-openai==0.1.23
langchain-text-splitters==0.2.4

Platform: Mac
Python version: 3.11.8

[dosubot[bot]]

To access the document metadata used by an agent in the middle of the processing when using LangChain, you can use the rag_chain_with_source setup. This setup retrieves the documents and their metadata, and then processes the question. Here is the relevant code snippet:

from operator import itemgetter
from langchain.schema.runnable import RunnableMap

rag_chain_from_docs = (
    {
        "context": lambda input: format_docs(input["documents"]),
        "question": itemgetter("question"),
    }
    | prompt
    | llm
    | StrOutputParser()
)
rag_chain_with_source = RunnableMap(
    {"documents": retriever, "question": RunnablePassthrough()}
) | {
    "documents": lambda input: [doc.metadata for doc in input["documents"]],
    "answer": rag_chain_from_docs,
}

rag_chain_with_source.invoke("<your question>")

This code will return the retrieved documents or certain source metadata from the documents along with the answer to your question.

Additionally, if you are using a model that includes response metadata, you can access this metadata via the AIMessage.response_metadata attribute. Here is an example of how to access response metadata for different model providers:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4-turbo")
msg = llm.invoke([("human", "What's the oldest known example of cuneiform")])
print(msg.response_metadata)

This approach can be adapted for other model providers such as Anthropic, Google VertexAI, Bedrock, MistralAI, Groq, TogetherAI, and FireworksAI by using their respective libraries and models.

For example, when using FireworksAI, you can set up the agent and retriever as follows:

from langchain_community.chat_models.fireworks import ChatFireworks
from langchain_core.retrievers import BaseRetriever
from langchain_core.documents import Document

class ArxivRetriever(BaseRetriever):
    def _get_relevant_documents(self, query: str, *, run_manager):
        # Implementation to retrieve documents from Arxiv
        pass

arxiv_tool = create_retriever_tool(ArxivRetriever(), "arxiv", "Arxiv retriever description")
tools = [arxiv_tool]

llm = ChatFireworks(model="accounts/fireworks/models/mixtral-8x7b-instruct", model_kwargs={"temperature": 0, "max_tokens": 2048, "top_p": 1}, cache=True)
prompt = hub.pull("hwchase17/react-json").partial(tools=render_text_description(tools), tool_names=", ".join([t.name for t in tools]))

agent = (
    {
        "input": lambda x: x["input"],
        "agent_scratchpad": lambda x: format_log_to_str(x["intermediate_steps"]),
    }
    | prompt
    | llm.bind(stop=["\nObservation"])
    | ReActJsonSingleInputOutputParser()
)

agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True, handle_parsing_errors=True).with_types(input_type=InputType)

This setup allows you to retrieve and process document metadata during the agent's execution.

For a more detailed example, you can refer to the LangChain documentation on citations, which demonstrates how to create a chain that returns both the answer and the retrieved documents:

from operator import itemgetter
from typing import List
from langchain_core.documents import Document
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import (
    RunnableLambda,
    RunnableParallel,
    RunnablePassthrough,
)

def format_docs(docs: List[Document]) -> str:
    formatted = [
        f"Article Title: {doc.metadata['title']}\nArticle Snippet: {doc.page_content}"
        for doc in docs
    ]
    return "\n\n" + "\n\n".join(formatted)

format = itemgetter("docs") | RunnableLambda(format_docs)
answer = prompt | llm | StrOutputParser()
chain = (
    RunnableParallel(question=RunnablePassthrough(), docs=wiki)
    .assign(context=format)
    .assign(answer=answer)
    .pick(["answer", "docs"])
)

chain.invoke("How fast are cheetahs?")

This example shows how to format the retrieved documents and include their metadata in the processing chain ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[eduardoscamargo]

Is the code above compatible with Langgraph? It adds additional steps and complexity just to get the metadata. How do I apply this code to a StateGraph?

[dosubot[bot]]

Yes, the code is compatible with LangGraph. To apply this code to a StateGraph to get the metadata, you can follow these steps:

1. Define and Compile the StateGraph:

import pprint
from langgraph.graph import END, StateGraph

workflow = StateGraph(GraphState)

# Define the nodes
workflow.add_node("retrieve", retrieve)  # retrieve
workflow.add_node("grade_documents", grade_documents)  # grade documents
workflow.add_node("generate", generate)  # generate
workflow.add_node("transform_query", transform_query)  # transform_query
workflow.add_node("web_search", web_search)  # web search

# Build graph
workflow.set_entry_point("retrieve")
workflow.add_edge("retrieve", "grade_documents")
workflow.add_conditional_edges(
    "grade_documents",
    decide_to_generate,
    {
        "transform_query": "transform_query",
        "generate": "generate",
    },
)
workflow.add_edge("transform_query", "web_search")
workflow.add_edge("web_search", "generate")
workflow.add_edge("generate", END)

# Compile
app = workflow.compile()

2. Run the Graph and Get the Metadata:

# Run
inputs = {"keys": {"question": "Explain how the different types of agent memory work?"}}
for output in app.stream(inputs):
    for key, value in output.items():
        pprint.pprint(f"Output from node '{key}':")
        pprint.pprint("---")
        pprint.pprint(value["keys"], indent=2, width=80, depth=None)
    pprint.pprint("\n---\n")

This will execute the StateGraph and print the metadata at each node ^[1].

For more detailed examples and additional features, you can refer to the LangGraph documentation, which includes guides on building with LangGraph and implementing workflows such as map-reduce. Additionally, you can use the metadata tagger document transformer to extract metadata from a LangChain Document, which covers similar functionality as the tagging chain but applied to a LangChain Document ^[2][3][4].

To continue talking to Dosu, mention @dosu.

[eduardoscamargo]

Yes, the code is compatible with LangGraph. To apply this code to a StateGraph to get the metadata, you can follow these steps:

1. Define and Compile the StateGraph:

import pprint
from langgraph.graph import END, StateGraph

workflow = StateGraph(GraphState)

# Define the nodes
workflow.add_node("retrieve", retrieve)  # retrieve
workflow.add_node("grade_documents", grade_documents)  # grade documents
workflow.add_node("generate", generate)  # generate
workflow.add_node("transform_query", transform_query)  # transform_query
workflow.add_node("web_search", web_search)  # web search

# Build graph
workflow.set_entry_point("retrieve")
workflow.add_edge("retrieve", "grade_documents")
workflow.add_conditional_edges(
    "grade_documents",
    decide_to_generate,
    {
        "transform_query": "transform_query",
        "generate": "generate",
    },
)
workflow.add_edge("transform_query", "web_search")
workflow.add_edge("web_search", "generate")
workflow.add_edge("generate", END)

# Compile
app = workflow.compile()

2. Run the Graph and Get the Metadata:

# Run
inputs = {"keys": {"question": "Explain how the different types of agent memory work?"}}
for output in app.stream(inputs):
    for key, value in output.items():
        pprint.pprint(f"Output from node '{key}':")
        pprint.pprint("---")
        pprint.pprint(value["keys"], indent=2, width=80, depth=None)
    pprint.pprint("\n---\n")

This will execute the StateGraph and print the metadata at each node [1].

For more detailed examples and additional features, you can refer to the LangGraph documentation, which includes guides on building with LangGraph and implementing workflows such as map-reduce. Additionally, you can use the metadata tagger document transformer to extract metadata from a LangChain Document, which covers similar functionality as the tagging chain but applied to a LangChain Document [2][3][4].

To continue talking to Dosu, mention @dosu.

This doesn't solve my problem