AmazonKendraRetriever forces to have all the fields required

[smg6135]

After the update of langchain_community, internally we use pydantic 2.X. Pydantic 2.X seems to force all fields required unlike Pydantic 1, which causes an error when invoke(query) on the retriever. What would be the recommended fix?

[dosubot[bot]]

Hey @smg6135! I'm here to help you with any bugs, questions, or contributions you have. Let's work together to solve this issue.

To make fields optional in Pydantic 2.X, you can use Optional from the typing module or set a default value for the fields. For the AmazonKendraRetriever, you can ensure that fields you want to be optional are defined with Optional or have default values. Here's an example of how you might define an optional field:

from typing import Optional
from pydantic import BaseModel

class AmazonKendraRetriever(BaseModel):
    index_id: str
    region_name: Optional[str] = None
    credentials_profile_name: Optional[str] = None
    # other fields...

In this example, region_name and credentials_profile_name are optional because they are defined with Optional and have a default value of None. This should help avoid errors related to missing fields when using Pydantic 2.X.

To continue talking to Dosu, mention @dosu.

[smg6135]

@dosu

File "/kendra_bedrock.py", line 35, in
response = retriever._kendra_query(query) # Get raw response
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/langchain_community/retrievers/kendra.py", line 441, in _kendra_query
r_result = RetrieveResult.parse_obj(response)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/pydantic/main.py", line 1162, in parse_obj
return cls.model_validate(obj)
^^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/pydantic/main.py", line 596, in model_validate
return cls.pydantic_validator.validate_python(

[dosubot[bot]]

Could you share the definition of the RetrieveResult class and any specific fields that are causing the issue? Additionally, if there are any error messages or stack traces beyond what you've provided, that information would be helpful as well.

To continue talking to Dosu, mention @dosu.

[smg6135]

@dosu

this is the full error messages. It's saying I need every single fields on the result items from Kendra

ResultItems.0.DocumentAttributes.0.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.0.DocumentAttributes.0.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.0.DocumentAttributes.0.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.0.DocumentAttributes.1.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.0.DocumentAttributes.1.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.0.DocumentAttributes.1.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.1.DocumentAttributes.0.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.1.DocumentAttributes.0.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.1.DocumentAttributes.0.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.1.DocumentAttributes.1.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.1.DocumentAttributes.1.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.1.DocumentAttributes.1.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.2.DocumentAttributes.0.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.2.DocumentAttributes.0.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.2.DocumentAttributes.0.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.2.DocumentAttributes.1.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.2.DocumentAttributes.1.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.2.DocumentAttributes.1.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.3.DocumentAttributes.0.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.3.DocumentAttributes.0.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.3.DocumentAttributes.0.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.3.DocumentAttributes.1.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.3.DocumentAttributes.1.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.3.DocumentAttributes.1.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.4.DocumentAttributes.0.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.4.DocumentAttributes.0.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.4.DocumentAttributes.0.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.4.DocumentAttributes.1.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.4.DocumentAttributes.1.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.4.DocumentAttributes.1.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.5.DocumentAttributes.0.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.5.DocumentAttributes.0.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.5.DocumentAttributes.0.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.5.DocumentAttributes.1.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.5.DocumentAttributes.1.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.5.DocumentAttributes.1.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.6.DocumentAttributes.0.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.6.DocumentAttributes.0.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.6.DocumentAttributes.0.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.6.DocumentAttributes.1.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.6.DocumentAttributes.1.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.6.DocumentAttributes.1.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.7.DocumentAttributes.0.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.7.DocumentAttributes.0.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.7.DocumentAttributes.0.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.7.DocumentAttributes.1.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.7.DocumentAttributes.1.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.7.DocumentAttributes.1.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.8.DocumentAttributes.0.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.8.DocumentAttributes.0.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.8.DocumentAttributes.0.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.8.DocumentAttributes.1.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.8.DocumentAttributes.1.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.8.DocumentAttributes.1.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.9.DocumentAttributes.0.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.9.DocumentAttributes.0.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.9.DocumentAttributes.0.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'https://...leaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.9.DocumentAttributes.1.Value.DateValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.9.DocumentAttributes.1.Value.LongValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing
ResultItems.9.DocumentAttributes.1.Value.StringListValue
Field required [type=missing, input_value={'StringValue': 'cleaned_ingredients.csv'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.9/v/missing

[dosubot[bot]]

To address the issue with Pydantic 2.X requiring all fields to be present in the RetrieveResult class, you can make fields optional by using Optional from the typing module and providing default values. This approach ensures that missing fields do not cause validation errors. Here's an example of how you can define the RetrieveResult class to handle optional fields:

from pydantic import BaseModel, Field
from typing import List, Optional

class DocumentAttributeValue(BaseModel):
    DateValue: Optional[str] = None
    LongValue: Optional[int] = None
    StringListValue: Optional[List[str]] = None
    StringValue: Optional[str] = None

class DocumentAttribute(BaseModel):
    Value: DocumentAttributeValue

class RetrieveResultItem(BaseModel):
    DocumentAttributes: List[DocumentAttribute] = Field(default_factory=list)

class RetrieveResult(BaseModel):
    ResultItems: List[RetrieveResultItem] = Field(default_factory=list)

In this setup:

• DateValue, LongValue, StringListValue, and StringValue in DocumentAttributeValue are marked as Optional and given default values of None.
• DocumentAttributes in RetrieveResultItem and ResultItems in RetrieveResult are initialized with empty lists using Field(default_factory=list), ensuring they are always present even if no items are returned.

This configuration should help you avoid errors related to missing fields when using Pydantic 2.X ^[1].

To continue talking to Dosu, mention @dosu.