How to pass other input to an evaluator in LangSmith?

[MichaelMMeskhi]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

N/A

Description

Given a question/PDF file set how do I pass the same PDF file to the evaluator LLM?

System Info

langchain==0.2.6
langchain-chroma==0.1.2
langchain-community==0.2.1
langchain-core==0.2.23
langchain-google-genai==1.0.7
langchain-google-vertexai==1.0.7
langchain-openai==0.1.13
langchain-text-splitters==0.2.0
langchainhub==0.1.20
openinference-instrumentation-langchain==0.1.22

[dosubot[bot]]

To pass a PDF file to the evaluator LLM in LangSmith, you need to load the PDF into a format that the LLM can handle using a document loader. Here is a step-by-step guide:

1. Install necessary packages:

   !pip install -U langchain openai chromadb langchain-experimental
   !pip install "unstructured[all-docs]==0.10.19" pillow pydantic lxml pillow matplotlib tiktoken open_clip_torch torch

2. Load the PDF using PyPDFLoader:

   from langchain_community.document_loaders import PyPDFLoader
   
   file_path = "path_to_your_pdf_file.pdf"  # Replace with your PDF file path
   loader = PyPDFLoader(file_path)
   
   pdf_pages = loader.load()

3. Split the loaded PDF content:

   from langchain_text_splitters import RecursiveCharacterTextSplitter
   
   text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)
   all_splits_pypdf = text_splitter.split_documents(pdf_pages)
   all_splits_pypdf_texts = [d.page_content for d in all_splits_pypdf]

4. Prepare the loaded documents for retrieval:

   from langchain_community.vectorstores import Chroma
   from langchain_openai import OpenAIEmbeddings
   
   baseline = Chroma.from_texts(
       texts=all_splits_pypdf_texts,
       collection_name="baseline",
       embedding=OpenAIEmbeddings(),
   )
   retriever_baseline = baseline.as_retriever()

5. Create the retrieval-augmented generation (RAG) chain:

   from langchain_core.prompts import ChatPromptTemplate
   from langchain_openai import ChatOpenAI
   from langchain_core.output_parsers import StrOutputParser
   from langchain_core.runnables import RunnablePassthrough
   
   template = """Answer the question based only on the following context, which can include text and tables:
   {context}
   Question: {question}
   """
   rag_prompt_text = ChatPromptTemplate.from_template(template)
   
   def text_rag_chain(retriever):
       model = ChatOpenAI(temperature=0, model="gpt-4")
       chain = (
           {"context": retriever, "question": RunnablePassthrough()}
           | rag_prompt_text
           | model
           | StrOutputParser()
       )
       return chain
   
   chain_baseline = text_rag_chain(retriever_baseline)

6. Run the evaluation:

   from langsmith import Client
   from langchain.smith import RunEvalConfig
   
   client = Client()
   eval_config = RunEvalConfig(evaluators=["qa"])
   
   def run_eval(chain, run_name, dataset_name):
       _ = client.run_on_dataset(
           dataset_name=dataset_name,
           llm_or_chain_factory=lambda: (lambda x: x["question"]) | chain,
           evaluation=eval_config,
           project_name=run_name,
       )
   
   dataset_name = "Your Dataset Name"
   run_eval(chain_baseline, dataset_name + "-baseline", dataset_name)

By following these steps, you can pass a PDF file to the evaluator LLM in LangSmith and perform question-answering tasks on the content of the PDF ^[1][2].

To continue talking to Dosu, mention @dosu.

[MichaelMMeskhi]

@dosu I am not using a RAG approach. I provide the PDF files via a URL for the LLM.

[dosubot[bot]]

To pass a PDF file provided via a URL to the evaluator LLM without using a RAG approach, you can use the OnlinePDFLoader to load the PDF and then directly pass the content to the LLM. Here is a step-by-step guide:

1. Install necessary packages:

   pip install langchain openai unstructured[all-docs] requests

2. Load the PDF using OnlinePDFLoader:

   from langchain_community.document_loaders import OnlinePDFLoader
   
   url = "https://arxiv.org/pdf/2302.03803.pdf"  # Replace with your PDF URL
   loader = OnlinePDFLoader(url)
   
   pdf_pages = loader.load()

3. Prepare the content for the LLM:

   all_text = "\n".join([page.page_content for page in pdf_pages])

4. Pass the content to the LLM:

   from langchain_openai import ChatOpenAI
   
   model = ChatOpenAI(temperature=0, model="gpt-4")
   
   question = "Your question here"  # Replace with your question
   response = model({"context": all_text, "question": question})
   
   print(response)

This approach directly loads the PDF content from the URL and passes it to the LLM for evaluation without using a retrieval-augmented generation (RAG) approach. The OnlinePDFLoader ensures that the PDF is fetched and processed correctly ^[1].

To continue talking to Dosu, mention @dosu.