Capture output of pandas dataframe agent

[hepbc]

Hi - I am using the langchain pandas dataframe agent. Somehow it is not returning the output in the output variable. It just displays the answer and output variable says "as shown in the observation above". I need the output in structured format to use it for further processing. Request help from the community. Code:

df1 = pd.read_excel("niftywtsjun24.xlsx", sheet_name="Jun24", skiprows=2)
df2 = pd.read_excel("niftywtsjun24.xlsx", sheet_name="Sep24", skiprows=2)
agent = create_pandas_dataframe_agent(OpenAI(temperature=0), [df1, df2], verbose=True, allow_dangerous_code=True, max_iterations = 5)
res = agent.invoke("what is the difference in weightages for each sector?")

Output:

Thought: I need to find the weightage for each sector in both dataframes and then calculate the difference.
Action: python_repl_ast
Action Input: df1.groupby('Sector')['Weightage (%)'].sum()Sector
DISCRETIONARY    11.23
ENERGY           14.66
FINANCIALS       34.43
HEALTHCARE        4.15
INDUSTRIALS       5.71
MATERIALS         6.36
STAPLES           7.29
TECH             12.53
TELECOM           3.64
Name: Weightage (%), dtype: float64I need to do the same for df2 and then calculate the difference between the two dataframes.
Action: python_repl_ast
Action Input: df2.groupby('Sector')['Weightage (%)'].sum()Sector
DISCRETIONARY    12.845493
ENERGY           13.444370
FINANCIALS       32.918292
HEALTHCARE        3.918168
INDUSTRIALS       6.170959
MATERIALS         6.065065
STAPLES           7.934651
TECH             12.749400
TELECOM           3.953603
Name: Weightage (%), dtype: float64 Now I need to calculate the difference between the two dataframes.
Action: python_repl_ast
Action Input: df1.groupby('Sector')['Weightage (%)'].sum() - df2.groupby('Sector')['Weightage (%)'].sum()Sector
DISCRETIONARY   -1.615493
ENERGY           1.215630
FINANCIALS       1.511708
HEALTHCARE       0.231832
INDUSTRIALS     -0.460959
MATERIALS        0.294935
STAPLES         -0.644651
TECH            -0.219400
TELECOM         -0.313603
Name: Weightage (%), dtype: float64 I now know the final answer.
Final Answer: The difference in weightages for each sector is shown above.

How do I capture the full table as output instead of just the text saying shown above.

Thanks!

[dosubot[bot]]

I found a similar unsolved discussion that might be relevant to your issue: Output of pandas dataframe agent, which was last updated on October 14, 2024 ^[1].

Regarding your question, the LangChain pandas dataframe agent typically returns outputs in a textual summary format. To capture the output in a structured format, you might need to modify the agent's behavior or the way it processes outputs. Unfortunately, specific methods or attributes to directly capture structured data weren't found in the available information. You might consider exploring the agent's internal methods or modifying the invoke method to capture the structured data you need.

To continue talking to Dosu, mention @dosu.