Inconsistent Structured Output with GPT-4o Despite temperature=0 and top_p=0 (AzureChatOpenAI)

[dip9811111]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_openai import AzureChatOpenAI
from pydantic import BaseModel, Field
from typing import Optional

class PydanticOfferor(BaseModel):
    name: Optional[str] = Field(description="Name of the company that makes the offer.")
    legal_address: Optional[str] = Field(description="Legal address of the company.")
    contact_people: Optional[List[str]] = Field(description="Contact people of the company")

class PydanticFinalReport(BaseModel):
    offeror: Optional[PydanticOfferor] = Field(description="Company making the offer.")
    language: Optional[str] = Field(description="Language of the document.")


MODEL = AzureChatOpenAI(
    azure_deployment=AZURE_MODEL_NAME,
    azure_endpoint=AZURE_ENDPOINT,
    api_version=AZURE_API_VERSION,
    temperature=0,
    top_p=0,
    max_tokens=None,
    timeout=None,
    max_retries=1,
    seed=42,
)

# Load document content
total_text = ""
for doc_path in docs_path:
    with open(doc_path, "r") as f:
        total_text += f"{f.read()}\n\n"

# Prompt
user_message = f"""Here is the report that you have to process:
[START REPORT]
{total_text}
[END REPORT]"""

messages = [
    {"role": "system", "content": self.system_prompt},
    {"role": "user", "content": user_message},
]

structured_llm = MODEL.with_structured_output(PydanticFinalReport, method="function_calling")
final_report_answer = structured_llm.invoke(messages)

Description

Hi all,

I'm currently using AzureChatOpenAI with the GPT-4o model and aiming to obtain structured output. To ensure deterministic behavior, I’ve explicitly set both temperature=0 and top_p=0. I've also fixed seed=42. However, I’ve noticed that the output is not always consistent.

Sometimes the variations are minor—for example, if the document clearly lists "John Doe" and "Jane Smith" as contact people, the model might correctly extract both names in one run, but in another run, it might only return "John Doe", or even re-order the names. While these differences are relatively subtle, they still suggest some nondeterminism. However, in other cases, the discrepancies are more significant—for instance, I’ve seen the model extract entirely unrelated names from elsewhere in the document, such as "Michael Brown", who is not listed as a contact person at all. This kind of inconsistent behavior is especially confusing given that the input and parameters and context remain unchanged.

Has anyone else observed this behavior with GPT-4o on Azure?

I'd love to understand:

• Is this expected behavior for GPT-4o?
• Could there be an internal randomness even with these parameters?
• Are there any recommended workarounds to force full determinism for structured outputs?

Thanks in advance for any insights!

System Info

azure-ai-documentintelligence==1.0.2
azure-ai-evaluation==1.5.0
azure-core==1.33.0
gradio==5.15.0
langchain==0.3.25
langchain-chroma==0.2.3
langchain-community==0.3.5
langchain-huggingface==0.1.2
langchain-ollama==0.2.3
langchain-openai==0.3.16
langgraph==0.2.71
marker-pdf==1.6.1
nltk==3.9.1
openpyxl==3.1.5
pydantic==2.9.2
python-docx==1.1.2
python-docx-replace==0.4.4
rank_bm25==0.2.2
weasyprint==65.0