How to get confidence score and with openai model with structured output

[AsmaaMHadir]

When I invoke an openai model without structured output, I have no problem getting its confidence score:

    def calculate_weighted_confidence_score(llm_results_metadata, top_n=5, weight_top=0.7, weight_all=0.3):
        # Extract log probabilities from the metadata (filtering out any None values)
        log_probs = [item['logprob'] for item in llm_results_metadata['logprobs']['content'] if item['logprob'] is not None]
        
        # If no log_probs are available, return 0 confidence
        if not log_probs:
            return 0.0

        # Convert log probabilities to probabilities
        probabilities = np.exp(log_probs)
        
        
        # Geometric mean of all token probabilities:
        geo_mean_all = np.exp(np.mean(log_probs))
        
        # Sort probabilities in descending order and take the top N
        top_probs = sorted(probabilities, reverse=True)[:top_n]
        # Geometric mean of the top tokens
        geo_mean_top = np.prod(top_probs) ** (1/len(top_probs))
        
        # Combine the two geometric means using the specified weights
        weighted_confidence = (weight_top * geo_mean_top) + (weight_all * geo_mean_all)
        
        # Convert to percentage
        return round(weighted_confidence * 100, 2)

llm = AzureChatOpenAI(
                azure_endpoint=LangGraphConfig.AZURE_CHAT_ENDPOINT,
                api_key=LangGraphConfig.AZURE_API_KEY,
                api_version=LangGraphConfig.AZURE_API_VERSION,
                streaming=False,
                logprobs=True,
                temperature=0.0
            )

response = llm.invoke(question)

confidencescore =     calculate_weighted_confidence_score(response.response_metadata)

however, if I run the llm with structured output like in here:

llm_w_structured_output = llm.with_structured_output(PydanticClass)

response = llm_w_structured_output.invoke(question)

I have no idea how to access the response metadata to get the logprobs and calculate the confidence score

[dosubot[bot]]

To access the response metadata and get the log probabilities when using an OpenAI model with structured output in LangChain, you can use the response_metadata attribute. Here's how you can modify your code to access the log probabilities:

# Assuming PydanticClass is defined elsewhere
llm_w_structured_output = llm.with_structured_output(PydanticClass)

response = llm_w_structured_output.invoke(question)

# Access the response metadata to get log probabilities
log_probs = response.response_metadata["logprobs"]["content"]

# Calculate the confidence score using your existing function
confidence_score = calculate_weighted_confidence_score(response.response_metadata)

The response_metadata attribute contains the log probabilities, which you can use to calculate the confidence score as you did with the non-structured output ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}