docs/concepts/structured_outputs/

[giscus[bot]]

docs/concepts/structured_outputs/

Overview

https://python.langchain.com/docs/concepts/structured_outputs/

[SharoMonk]

model = ChatGroq(model="llama3-8b-8192")
model_with_structured_output = model.with_structured_output(schema=ResponseFormatter)

Invoke the model to produce structured output that matches the schema

structured_output = model_with_structured_output.invoke(
"My question is why did you do it and how was it done?"
)
print(structured_output)
answer='Why did you do it?' followup_question='How was it done?'

[passaadellism]

Can we return the response metadata that are contained to the simple invoke eg (AIMessage) with the structured schema?

response_metadata={'token_usage': {'completion_tokens': 107, 'prompt_tokens': 409, 'total_tokens': 516, 'completion_tokens_details': {'reasoning_tokens': 0, 'audio_tokens': 0, 'accepted_prediction_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'cached_tokens': 0, 'audio_tokens': 0}}, 'model_name': 'gpt-4o-2024-08-06', 'system_fingerprint': 'fp_xxxxxxxx', 'finish_reason': 'stop', 'logprobs': None}, id='run-xxxxxxxxxxxxxx', usage_metadata={'input_tokens': 409, 'output_tokens': 107, 'total_tokens': 516}

[syntaxtrash]

I'm not sure if I understand your question. Are you asking how to get the response_metadata attribute from AIMessage?

You can access it directly like this:
response.response_metadata

[Arghya721]

Using a prompt template along with tool calling is giving me an error. Also it seems the llm is not able to read the input prompt provided.

import dotenv
import os
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.messages import HumanMessage, SystemMessage
from pydantic import BaseModel, Field

dotenv.load_dotenv()

class ResponseFormatter(BaseModel):
    """Always use this tool to structure your response to the user."""
    language: str = Field(description="The language of the translation")
    text: str = Field(description="The text to be translated")
    translation: str = Field(description="The translation of the text")

open_ai = ChatOpenAI(
    model="gpt-4o-mini"
)

open_ai_tools = open_ai.bind_tools([ResponseFormatter])

system_template = """
Translate the following from English into {language}:
"""

prompt_template = ChatPromptTemplate.from_messages(
    [("system", system_template), ("user", "{text}")]
)

prompt = prompt_template.invoke({"language" : "German", "text" : "hi!"})

print(prompt)

ai_msg = open_ai_tools.invoke(prompt)

print(ai_msg)

print(ai_msg.tool_calls[0]["args"])

pydantic_object = ResponseFormatter.model_validate(ai_msg.tool_calls[0]["args"])

print(pydantic_object)

Output

messages=[SystemMessage(content='\nTranslate the following from English into German:\n', additional_kwargs={}, response_metadata={}), HumanMessage(content='hi!', additional_kwargs={}, response_metadata={})]
content='Hello! How can I assist you today?' additional_kwargs={'refusal': None} response_metadata={'token_usage': {'completion_tokens': 11, 'prompt_tokens': 87, 'total_tokens': 98, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'fp_06737a9306', 'finish_reason': 'stop', 'logprobs': None} id='run-a67cb852-ad94-4923-95a1-89535e8f6591-0' usage_metadata={'input_tokens': 87, 'output_tokens': 11, 'total_tokens': 98, 'input_token_details': {'audio': 0, 'cache_read': 0}, 'output_token_details': {'audio': 0, 'reasoning': 0}}
Traceback (most recent call last):
  File "D:\langchain tutorial\structured_output.py", line 40, in <module>
    print(ai_msg.tool_calls[0]["args"])
          ~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

[SuperMuel]

This is the problem that they mention in here

(2) In addition, the model needs to be instructed to always use the tool when we want to enforce structured output, which is a provider specific setting.

The problem is that LLM answered with a classical AIMessage instead of using the tool you provided. There is 0 tool messages, hence the list index out of range error. The LLM has a tool, but doesn't know why it should use it. To solve this error, you have to instruct the LLM to always use the tool provided.

But the fast answer is : Just use the .with_structured_output method.

There might also be a prompt issue. Maybe changing the system prompt to something like You are a translator. Translate every message to German.

[authority-figures]

When I tried to use the with_structured_output (include_raw = True) method on the LLM after binding the tool, I found that the content returned by the LLM no longer contained toolcalls.

use with_structured_output

llm_with_structure_output = llm_with_tools.with_structured_output(CustomOutput,include_raw=True)
chain = prompt_template | llm_with_structure_output

from pydantic import BaseModel, Field
from langchain.prompts import ChatPromptTemplate
from langchain_core.messages import HumanMessage
from langchain_core.messages.tool import ToolCall
from agent_project.agent.llm import chatGPT_llm
from agent_project.agent.tools import load_robot,move_robot_to_target,create_cube,clear_env

from langchain.output_parsers import PydanticOutputParser
from typing import List, Dict
from typing_extensions import Annotated, TypedDict

class CustomOutput(BaseModel):

    task:  Annotated[str, ..., "The task you are performing"]



prompt_template = ChatPromptTemplate.from_template(
    """You are an intelligent assistant who must use tools to solve problems。

{input}
"""
)

llm = chatGPT_llm(
    model_name="gpt-4o-mini", temperature=0
)

simulation_tools = [load_robot,move_robot_to_target,create_cube,clear_env]

llm_with_tools = llm.bind_tools(simulation_tools)
llm_with_structure_output = llm_with_tools.with_structured_output(CustomOutput,include_raw=True)

chain = prompt_template | llm_with_structure_output

# Run the chain and get the output
result = chain.invoke({"input": "Please help me add a robotic arm to the simulation environment at (1,1,0) with the file path F:\ sw\ urdf_files\ minicobo_v1\ urdf\ minicobo_v1. 4.urdf '.",})
print(result)

output

result['raw'].toolcalls = []
result['raw'].content = "To add a robotic arm to the simulation environment at the specified coordinates (1,1,0) using the provided URDF file path, ..."
result['parsed'].task = "To add a robotic arm to the simulation environment at the specified ..."

not use with_structured_output

chain = prompt_template | llm_with_tools

output

result.toolcalls = [{'name': 'load_robot', 'args': {'base_position': [1, 1, 0], 'description': 'Add a robotic arm to the simulation environment.', 'urdf_path': 'F:\\ sw\\ urdf_files\\ minicobo_v1\\ urdf\\ minicobo_v1. 4.urdf'}, 'id': 'call_jLjzcDgXAsOzSjFwleNMXzM2', 'type': 'tool_call'}]
result.content = ""

I would like to know why on the LLM bound to tools, using formatted output, the LLM will no longer call the tools. And how to solve it?
The purpose of this is to hope that when the LLM executes toolcalls, it can output a description of the execution process of the task together with the tool calls, which can help me record the process of the LLM executing the task