Issues with with_structured_output(...) in ChatVertexAI: KeyError and TypeError #18975
Replies: 6 comments 19 replies
-
|
Hi @jorge2najo, Can you try JSON mode ? Tell me what you get structured_llm = llm.with_structured_output(AnswerWithJustification, method="json_mode") |
Beta Was this translation helpful? Give feedback.
-
|
@jorge2najo from langchain_community.chat_models.vertexai import ChatVertexAI |
Beta Was this translation helpful? Give feedback.
-
|
Hi @maximeperrindev, thank you for your answer. When I run the script 1 with your suggestion I get this error: Error: Script 1: from langchain_core.pydantic_v1 import BaseModel
from langchain_community.chat_models.vertexai import ChatVertexAI
class AnswerWithJustification(BaseModel):
'''An answer to the user question along with justification for the answer.'''
answer: str
justification: str
llm = ChatVertexAI(model="gemini-pro", temperature=0)
structured_llm = llm.with_structured_output(AnswerWithJustification)
structured_llm.invoke("What weighs more a pound of bricks or a pound of feathers")Environment Details: |
Beta Was this translation helpful? Give feedback.
-
|
@jorge2najo from langchain.pydantic_v1 import BaseModel
from langchain_google_vertexai import create_structured_runnable
llm = ChatVertexAI(model_name="gemini-pro")
class MyModel(BaseModel):
name: str
age: int
chain = create_structured_runnable(MyModel, llm)
chain.invoke("My name is Erick and I'm 27 years old") |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately, even after importing create_structured_runnable, I still encounter an error when trying that code. Script from langchain_core.pydantic_v1 import BaseModel
#from langchain_community.chat_models.vertexai import ChatVertexAI
from langchain_google_vertexai import ChatVertexAI
from langchain_google_vertexai import create_structured_runnable
llm = ChatVertexAI(model="gemini-pro")
class MyModel(BaseModel):
name: str
age: int
chain = create_structured_runnable(MyModel, llm)
chain.invoke("My name is Erick and I'm 27 years old")Error: System Information
Package Information
|
Beta Was this translation helpful? Give feedback.
-
|
@jorge2najo from langchain_google_vertexai import VertexAI
llm = VertexAI(model_name="gemini-pro")It's probably not going to work but worth it to try |
Beta Was this translation helpful? Give feedback.
-
|
Hi @maximeperrindev, thank you again for your answer. Unfortunately it didn't work. Script: from langchain_core.pydantic_v1 import BaseModel
#from langchain_community.chat_models.vertexai import ChatVertexAI
#from langchain_google_vertexai import ChatVertexAI
from langchain_google_vertexai import create_structured_runnable
from langchain_google_vertexai import VertexAI
class AnswerWithJustification(BaseModel):
'''An answer to the user question along with justification for the answer.'''
answer: str
justification: str
llm = VertexAI(model="gemini-pro", temperature=0)
# structured_llm = llm.with_structured_output(AnswerWithJustification)
chain = create_structured_runnable(AnswerWithJustification, llm)
chain.invoke("What weighs more a pound of bricks or a pound of feathers")Error: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @maximeperrindev, just an update from last week. I made small changes in the script before and now it works: ChatVertexAI(model_name="gemini-pro") Script: from langchain.pydantic_v1 import BaseModel
from langchain_google_vertexai import ChatVertexAI
from langchain_google_vertexai import create_structured_runnable
from langchain_google_vertexai import VertexAI
llm = ChatVertexAI(model_name="gemini-pro")
class AnswerWithJustification(BaseModel):
'''An answer to the user question along with justification for the answer.'''
answer: str
justification: str
# structured_llm = llm.with_structured_output(AnswerWithJustification)
chain = create_structured_runnable(AnswerWithJustification, llm)
result = chain.invoke("What weighs more a pound of bricks or a pound of feathers")
print(result)System Information
Package Information
|
Beta Was this translation helpful? Give feedback.
-
|
I get an related error when following the example notebook The script below runs fine if the schema is set to However, it failed when the schema is set to runnable = prompt | llm.with_structured_output(schema=Data)Versionimport langchain
from google.cloud import aiplatform
print(f"LangChain version: {langchain.__version__}")
print(f"Vertex AI SDK version: {aiplatform.__version__}")LangChain version: 0.1.11 Scriptfrom typing import List, Optional
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_google_vertexai import ChatVertexAI, HarmCategory, HarmBlockThreshold
class Person(BaseModel):
"""Information about a person."""
name: Optional[str] = Field(..., description="The name of the person")
hair_color: Optional[str] = Field(
..., description="The color of the person's hair if known"
)
height_in_meters: Optional[str] = Field(
..., description="Height measured in meters"
)
class Data(BaseModel):
"""Extracted data about people."""
people: List[Person]
prompt = ChatPromptTemplate.from_messages(
[
(
"system",
"You are an expert extraction algorithm. "
"Only extract relevant information from the text. "
"If you do not know the value of an attribute asked to extract, "
"return null for the attribute's value.",
),
# Please see the how-to about improving performance with
# reference examples.
# MessagesPlaceholder('examples'),
("human", "{text}"),
]
)
llm = ChatVertexAI(
model_name="gemini-pro",
temperature=0,
safety_settings={
HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,
},
convert_system_message_to_human=True,
)
text = "My name is Jeff, my hair is black and i am 6 feet tall. Anna has the same color hair as me."
runnable = prompt | llm.with_structured_output(schema=Data)
runnable.invoke({"text": text})ErrorOther questionsIs there an analog function to import json
from langchain_core.utils.function_calling import convert_to_openai_tool
print(json.dumps(convert_to_openai_tool(Data), indent=2))Output: {
"type": "function",
"function": {
"name": "Data",
"description": "Extracted data about people.",
"parameters": {
"type": "object",
"properties": {
"people": {
"type": "array",
"items": {
"description": "Information about a person.",
"type": "object",
"properties": {
"name": {
"description": "The name of the person",
"type": "string"
},
"hair_color": {
"description": "The color of the person's hair if known",
"type": "string"
},
"height_in_meters": {
"description": "Height measured in meters",
"type": "string"
}
},
"required": [
"name",
"hair_color",
"height_in_meters"
]
}
}
},
"required": [
"people"
]
}
}
}The How to see the generated Schema used in the call of the tool? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @maximeperrindev, I have a similar question regarding There is a way to bind functions to a ChatVertexAI(...)? With the ChatOpenAI(...) the following script is possible: Script: llm_tool = ChatOpenAI(model="gpt-4-turbo-preview", temperature=0)
tools = [MyModel1, MyModel2, ...]
functions = [convert_to_openai_function(t) for t in tools]
model_tool = llm_tool.bind_functions(functions) |
Beta Was this translation helpful? Give feedback.
-
|
@jorge2najo from langchain.agents import AgentExecutor # type: ignore
from langchain.agents.format_scratchpad import ( # type: ignore
format_to_openai_function_messages,
)
from langchain.chains import LLMMathChain # type: ignore
llm = ChatVertexAI(model_name="gemini-pro")
math_chain = LLMMathChain.from_llm(llm=llm)
tools = [
Tool(
name="Calculator",
func=math_chain.run,
description="useful for when you need to answer questions about math",
)
]
prompt = ChatPromptTemplate.from_messages(
[
("user", "{input}"),
MessagesPlaceholder(variable_name="agent_scratchpad"),
]
)
llm_with_tools = llm.bind(functions=tools)
agent = (
{ # type: ignore
"input": lambda x: x["input"],
"agent_scratchpad": lambda x: format_to_openai_function_messages(
x["intermediate_steps"]
),
}
| prompt
| llm_with_tools
| _TestOutputParser()
)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
response = agent_executor.invoke({"input": "What is 6 raised to the 0.43 power?"})
print(response) |
Beta Was this translation helpful? Give feedback.
-
|
Hi @maximeperrindev, thank you for your answer. Unfortunately, the solution you provided didn't work for my scenario. What I am trying to achieve is for the LLM, in this case, ChatVertexAI(...), to output the 'arguments' and the 'function_name' from a list of tools. This objective has been my focus since the beginning of this issue, which is why I attempted to use 'with_structured_output()' or 'create_structured_runnable()'. Unfortunately, the attempt to use 'create_structured_runnable()' didn't work in the scenario outlined below. Do you know a way to achieve this using langchain modules? Script: (Note: This script doesn't work. It's only meant to illustrate what I'm aiming for) from tools_format_openai import*
from langchain_google_vertexai import ChatVertexAI
from langchain_core.utils.function_calling import convert_to_openai_function
import json
################################################################################################################
class AgentExecutorGemini:
def __init__(self, model, message):
self.model = model
self.message = message
def get_execution(self):
# Creation of the list of tools
tools = [WakeUp(), SayHello(), TakeBreakfast()]
llm = ChatVertexAI(model=self.model, temperature=0)
# Binding the tools to the model
functions = [convert_to_openai_function(t) for t in tools]
llm_with_tools = llm.bind(tools=functions)
result = llm_with_tools(self.message)
#### function_name ####
tool = result.additional_kwargs["function_call"]["name"]
print(tool)
#### arguments ####
tool_input = json.loads(result.additional_kwargs["function_call"]["arguments"])
print(tool_input)
#### THIS DOES NOT WORK #############################################
# from langchain_google_vertexai import create_structured_runnable
# tools = [WakeUp(), SayHello(), TakeBreakfast()]
# chain = create_structured_runnable(tools, llm)
# result = chain.invoke(self.message)
#####################################################################
if __name__ == '__main__':
message = "Lara, you have to wake up at 6:00 am."
model = "gemini-pro"
agent_executor_openai = AgentExecutorGemini(model, message)
agent_executor_openai.get_execution()The output should be: |
Beta Was this translation helpful? Give feedback.
-
|
@jorge2najo okay i get what you want. Originally, we were trying to make from typing import Optional
from langchain_google_vertexai import ChatVertexAI, create_structured_runnable
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.pydantic_v1 import BaseModel, Field
class RecordPerson(BaseModel):
"""Record some identifying information about a person."""
name: str = Field(..., description="The person's name")
age: int = Field(..., description="The person's age")
fav_food: Optional[str] = Field(None, description="The person's favorite food")
class RecordDog(BaseModel):
"""Record some identifying information about a dog."""
name: str = Field(..., description="The dog's name")
color: str = Field(..., description="The dog's color")
fav_food: Optional[str] = Field(None, description="The dog's favorite food")
llm = ChatVertexAI(model_name="gemini-pro")
prompt = ChatPromptTemplate.from_template("""
You are a world class algorithm for recording entities.
Make calls to the relevant function to record the entities in the following input: {input}
Tip: Make sure to answer in the correct format"""
)
chain = create_structured_runnable([RecordPerson, RecordDog], llm, prompt=prompt)
chain.invoke({"input": "Harry was a chubby brown beagle who loved chicken"})
# -> RecordDog(name="Harry", color="brown", fav_food="chicken")Can you just try to run this ? If it doesn't work, we will probably need to open an issue then |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @maximeperrindev Setting the optional Fields to None class Person(BaseModel):
"""Information about a person."""
name: Optional[str] = Field(None, description="The name of the person")
hair_color: Optional[str] = Field(
None, description="The color of the person's hair if known"
)
height_in_meters: Optional[str] = Field(
None, description="Height measured in meters"
)and running from langchain_google_vertexai import create_structured_runnable
text = "My name is Jeff, my hair is black and **i am 6 feet tall**. Anna has the same color hair as me."
chain = create_structured_runnable(Data, llm)
chain.invoke(text)returns But the persons However, running the following code still returns an error text = "My name is Jeff, my hair is black and i am 6 feet tall. Anna has the same color hair as me."
runnable = prompt | llm.with_structured_output(schema=Data)
runnable.invoke({"text": text})Error |
Beta Was this translation helpful? Give feedback.
-
|
What if you set class Person(BaseModel):
"""Information about a person."""
name: Optional[str] = Field(None, description="The name of the person")
hair_color: Optional[str] = Field(
None, description="The color of the person's hair if known"
)
height_in_feet: Optional[str] = Field(
None, description="Height measured in feet"
) |
Beta Was this translation helpful? Give feedback.
-
|
Hi @maximeperrindev, the script you suggested me works. 👍 Thank you. If for example I used 'create_structured_runnable' in the script 1 I get an error: Script 1: from typing import Optional, Type
from langchain_google_vertexai import ChatVertexAI, create_structured_runnable
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain.tools import tool, BaseTool
class RecordDog(BaseModel):
"""Record some identifying information about a dog."""
name: str = Field(..., description="The dog's name")
color: str = Field(..., description="The dog's color")
fav_food: Optional[str] = Field(None, description="The dog's favorite food")
class RecordDogTool(BaseTool):
name = "record_dog_tool"
description = "useful for when you need to record information about a dog"
args_schema: Type[BaseModel] = RecordDog
return_direct: bool = False
def _run(self, name: str = "Harry", color: str = "brown", fav_food: Optional[str] = "chicken"
) -> str:
"""Use the tool."""
return f"Recorded {name} who is {color} and loves {fav_food}."
class RecordPerson(BaseModel):
"""Record some identifying information about a person."""
name: str = Field(..., description="The person's name")
age: int = Field(..., description="The person's age")
fav_food: Optional[str] = Field(None, description="The person's favorite food")
class RecordPersonTool(BaseTool):
name = "record_person_tool"
description = "useful for when you need to record information about a person"
args_schema: Type[BaseModel] = RecordPerson
return_direct: bool = False
def _run(self, name: str = "Pedro", age: int = 30, fav_food: Optional[str] = "Pizza"
) -> str:
"""Use the tool."""
return f"Recorded {name} who is {age} years old and loves {fav_food}."
llm = ChatVertexAI(model_name="gemini-pro")
prompt = ChatPromptTemplate.from_template("""
You are a world class algorithm for recording entities.
Make calls to the relevant function to record the entities in the following input: {input}
Tip: Make sure to answer in the correct format"""
)
chain = create_structured_runnable([RecordDogTool, RecordPersonTool], llm, prompt=prompt)
result = chain.invoke({"input":"Martin is 30 years old who loves paella"})
print(result)Error: Problem: The problem is, I would like to use the following structure to get the 'function_name' and the 'arguments'. This works perfectly if I use ChatOpenAI. This information is relevant for me, because then I can use the module 'ToolExecutor' from langgraph.prebuilt.tool_executor. Maybe some idea for ChatVertexAI?: Script I would like to use: class RecordPerson(BaseModel):
"""Record some identifying information about a person."""
name: str = Field(..., description="The person's name")
age: int = Field(..., description="The person's age")
fav_food: Optional[str] = Field(None, description="The person's favorite food")
class RecordPersonTool(BaseTool):
name = "record_person_tool"
description = "useful for when you need to record information about a person"
args_schema: Type[BaseModel] = RecordPerson
return_direct: bool = False
def _run(self, name: str = "Pedro", age: int = 30, fav_food: Optional[str] = "Pizza"
) -> str:
"""Use the tool."""
return f"Recorded {name} who is {age} years old and loves {fav_food}."
# Creation of the list of tools
tools = [RecordPersonTool(), ..............]
llm = ChatVertexAI(model=self.model, temperature=0)
# Binding the tools to the model
functions = [convert_to_openai_function(t) for t in tools]
llm_with_tools = llm.bind(tools=functions)
result = llm_with_tools("Martin is 30 years old who loves paella")
#### function_name ####
tool = result.additional_kwargs["function_call"]["name"]
print(tool)
#### arguments ####
tool_input = json.loads(result.additional_kwargs["function_call"]["arguments"])
print(tool_input)
Expected output: |
Beta Was this translation helpful? Give feedback.
-
|
@jorge2najo the mistake here from langchain_core.pydantic_v1 import BaseModel
from langchain_google_vertexai import ChatVertexAI
class AnswerWithJustification(BaseModel):
'''An answer to the user question along with justification for the answer.'''
answer: str
justification: str
llm = ChatVertexAI(model="gemini-pro", temperature=0)
structured_llm = llm.with_structured_output(AnswerWithJustification)
structured_llm.invoke("What weighs more a pound of bricks or a pound of feathers")is you need to specify apologies for the issue, i know this mistake is from the docs. will update docs and also update ChatVertexAI to treat model and model_name the same |
Beta Was this translation helpful? Give feedback.
-
|
supporting |
Beta Was this translation helpful? Give feedback.
-
|
@baskaryan Oh wow the solution was easier than I thought lol |
Beta Was this translation helpful? Give feedback.
-
|
Hi @baskaryan and @maximeperrindev, it worked. Thank you. 👍 |
Beta Was this translation helpful? Give feedback.
-
|
glad to hear it works, sorry for the issue and thanks for flagging @jorge2najo @maximeperrindev! |
Beta Was this translation helpful? Give feedback.
-
|
I am getting the same and I have not mentioned the model name. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I'm encountering errors when attempting to use ChatVertexAI with the with_structured_output(...) method in two separate scripts. The first script results in a KeyError: 'type', and the second script throws a TypeError: _ChatModelBase.start_chat() got an unexpected keyword argument 'functions'.
Details:
Code Samples:
Script 1:
Script 1 Error: Error Message: KeyError: 'type'
Script 2:
Script 2 Error: Error Message: TypeError: _ChatModelBase.start_chat() got an unexpected keyword argument 'functions'
Environment Details:
I'm working with the following libraries:
Questions:
I appreciate any insights or suggestions the community can offer. Thank you!
Beta Was this translation helpful? Give feedback.
All reactions