Training an LLM for Natural Responses Based on Dataframe Insights

[mraguth]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

The code I am using:

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_experimental.tools import PythonAstREPLTool
from langchain.agents import AgentExecutor, OpenAIFunctionsAgent

class PythonInputs(BaseModel):
    query: str = Field(description="code snippet to run")

df = pd.read_csv("your_data.csv")
template = """
You are working with a pandas dataframe in Python. The name of the dataframe is `df`.
It is important to understand the attributes of the dataframe before working with it. This is the result of running `df.head().to_markdown()`

<df>
{dhead}
</df>

You are not meant to use only these rows to answer questions - they are meant as a way of telling you about the shape and schema of the dataframe.
You also do not have use only the information here to answer questions - you can run intermediate queries to do exploratory data analysis to give you more information as needed.

For example:

<question>How old is Jane?</question>
<logic>Use `person_name_search` since you can use the query `Jane`</logic>
<question>Who has id 320</question>
<logic>Use `python_repl` since even though the question is about a person, you don't know their name so you can't include it.</logic>
"""  # noqa: E501

template = template.format(dhead=df.head().to_markdown())

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", template),
        MessagesPlaceholder(variable_name="agent_scratchpad"),
        ("human", "{input}"),
    ]
)

repl = PythonAstREPLTool(
    locals={"df": df},
    name="python_repl",
    description="Runs code and returns the output of the final line",
    args_schema=PythonInputs,
)
tools = [repl]
agent = OpenAIFunctionsAgent(
    llm=ChatOpenAI(temperature=0, model="gpt-4"), prompt=prompt, tools=tools
)
agent_executor = AgentExecutor(
    agent=agent, tools=tools, max_iterations=5, early_stopping_method="generate"
) | (lambda x: x["output"])

# Typing for playground inputs
class AgentInputs(BaseModel):
    input: str

agent_executor = agent_executor.with_types(input_type=AgentInputs)

# Example usage
query = "What is the average age of passengers?"
response = agent_executor.invoke({"input": query})
print(response)

Description

I am using the PythonAstREPLTool to query the dataframe and gain insights, utilizing the suggested code below. My objective is to have the bot respond to end users in a natural manner. For instance, my table field is labeled 'pt-id'. However, instead of the bot saying '100 pt-ids', I want it to say '100 patients'. The question is, where and how can I train the model to generate meaningful insights from the data? In llama-index, we used to define this in the 'response synthesis prompt'. Is there a similar method to train the LLM about our dataframe? Please help.

System Info

Name: langchain
Version: 0.2.16

[dosubot[bot]]

I found a similar unsolved discussion that might be relevant to your question: Changing the template and adding customization in pandas_dataframe_agent of OpenAI(gpt-4 model) in langchain ^[1].

To train the model to generate meaningful insights from the dataframe and respond naturally, such as saying '100 patients' instead of '100 pt-ids', using the PythonAstREPLTool and LangChain, you can follow these steps:

1. Set up the environment and load the CSV data:

   import pandas as pd
   from langchain.agents import AgentExecutor, OpenAIFunctionsAgent
   from langchain_community.chat_models import ChatOpenAI
   from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
   from langchain_experimental.tools import PythonAstREPLTool
   from langchain_core.pydantic_v1 import BaseModel, Field
   from pathlib import Path
   
   MAIN_DIR = Path(__file__).parents[1]
   df = pd.read_csv(MAIN_DIR / "your_data.csv")

2. Create a prompt template to guide the LLM:

   TEMPLATE = """You are working with a pandas dataframe in Python. The name of the dataframe is `df`.
   It is important to understand the attributes of the dataframe before working with it. This is the result of running `df.head().to_markdown()`
   
   <df>
   {dhead}
   </df>
   
   You are not meant to use only these rows to answer questions - they are meant as a way of telling you about the shape and schema of the dataframe.
   You also do not have use only the information here to answer questions - you can run intermediate queries to do exploratory data analysis to give you more information as needed.
   
   When providing answers, ensure they are complete sentences and represent numbers in a readable format, such as '100 patients' instead of '100 pt-ids'.
   
   For example:
   
   <question>How old is Jane?</question>
   <logic>Use `person_name_search` since you can use the query `Jane`</logic>
   <question>Who has id 320</question>
   <logic>Use `python_repl` since even though the question is about a person, you don't know their name so you can't include it.</logic>
   """  # noqa: E501
   
   template = TEMPLATE.format(dhead=df.head().to_markdown())
   
   prompt = ChatPromptTemplate.from_messages(
       [
           ("system", template),
           MessagesPlaceholder(variable_name="agent_scratchpad"),
           ("human", "{input}"),
       ]
   )

3. Set up the PythonAstREPLTool and other tools:

   class PythonInputs(BaseModel):
       query: str = Field(description="code snippet to run")
   
   repl = PythonAstREPLTool(
       locals={"df": df},
       name="python_repl",
       description="Runs code and returns the output of the final line",
       args_schema=PythonInputs,
   )
   tools = [repl]

4. Combine the prompt with the LLM and tools:

   agent = OpenAIFunctionsAgent(
       llm=ChatOpenAI(temperature=0, model="gpt-4"), prompt=prompt, tools=tools
   )
   agent_executor = AgentExecutor(
       agent=agent, tools=tools, max_iterations=5, early_stopping_method="generate"
   ) | (lambda x: x["output"])

5. Invoke the chain with a question and get a natural response:

   class AgentInputs(BaseModel):
       input: str
   
   agent_executor = agent_executor.with_types(input_type=AgentInputs)
   
   response = agent_executor.invoke({"input": "How many patients are there in the dataset?"})
   print(response)

This setup will allow the model to generate Python code to analyze the dataframe and provide a natural language response. By modifying the prompt template to include instructions for natural language responses, you can guide the model to produce more human-like answers ^[2][3][4].

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu can I update my dataframe fields and it's descriptions in the 'For Example: session? what is the significance of that part?

For example:

<question>How old is Jane?</question>
<logic>Use `person_name_search` since you can use the query `Jane`</logic>
<question>Who has id 320</question>
<logic>Use `python_repl` since even though the question is about a person, you don't know their name so you can't include it.</logic>
"""  # noqa: E501

[dosubot[bot]]

Yes, you can update your dataframe fields and their descriptions in the "For Example:" section. This part of the template provides the LLM with examples of how to handle specific types of queries and the logic it should use to answer them. By customizing this section, you can guide the LLM to understand the context and the appropriate methods to use for different types of questions.

Here's an updated example based on your dataframe fields:

template = """
You are working with a pandas dataframe in Python. The name of the dataframe is `df`.
It is important to understand the attributes of the dataframe before working with it. This is the result of running `df.head().to_markdown()`

<df>
{dhead}
</df>

You are not meant to use only these rows to answer questions - they are meant as a way of telling you about the shape and schema of the dataframe.
You also do not have use only the information here to answer questions - you can run intermediate queries to do exploratory data analysis to give you more information as needed.

When providing answers, ensure they are complete sentences and represent numbers in a readable format, such as '100 patients' instead of '100 pt-ids'.

For example:

<question>How old is Jane?</question>
<logic>Use `df[df['name'] == 'Jane']['age'].values[0]` to find Jane's age.</logic>
<question>Who has id 320?</question>
<logic>Use `df[df['id'] == 320]['name'].values[0]` to find the name of the person with id 320.</logic>
<question>What is the average age of patients?</question>
<logic>Use `df['age'].mean()` to calculate the average age of patients.</logic>
"""  # noqa: E501

template = template.format(dhead=df.head().to_markdown())

In this example, the <question> and <logic> pairs provide the LLM with clear instructions on how to handle specific queries related to your dataframe. This helps the LLM understand the context and generate more accurate and natural responses.

To continue talking to Dosu, mention @dosu.