How to calculate the cost of calling an LLM with structured output

[AsmaaMHadir]

I have the following node in my LangGraph graph:

async def generate_answer(self, state: State):
       """Generate the final answer based on context."""
       formatted_context = LangGraphUtils.format_context(state["context"])
       prompt_input = {"question": state["messages"][-1].content,"context": formatted_context,"chat_history": state["messages"]
        }
       out_prompt = ChatPromptTemplate.from_template(generation_prompt_template_v3).invoke(prompt_input)
       try:
            from langchain_community.callbacks.openai_info import OpenAICallbackHandler
            cb = OpenAICallbackHandler()

            response = await LangGraphConfig.initialize_llm().with_structured_output(GeneratedAnswer,method="function_calling").ainvoke(out_prompt)

            confidence_score = LangGraphUtils.calculate_weighted_confidence_score(response.response_metadata)
            state["answer"] = response.content or "Could not generate an answer."
            state["confidence_score"] = confidence_score
            return {"answer": state["answer"], "context": state["context"],"confidence_score": state["confidence_score"], "input_tokens": cb.prompt_tokens, "output_tokens": cb.completion_tokens, "total_tokens": cb.total_tokens, "cost": state["cost"] + cb.total_cost}
       except Exception:
            logger.error("Error generating answer:\n" + traceback.format_exc())
            return None

I tried using with get_openai_callback() as CB to calculate the cost and get the tokens count: but I get an error since I am using the llm with structured output

[dosubot[bot]]

To calculate the cost of calling an LLM with structured output in your LangGraph graph, you can use the OpenAICallbackHandler to track token usage and cost. Here's how you can modify your code to achieve this:

1. Import the OpenAICallbackHandler.
2. Initialize the callback handler before making the LLM call.
3. Use the callback handler to access token usage and cost after the LLM call.

Here's an example based on your provided code:

async def generate_answer(self, state: State):
    """Generate the final answer based on context."""
    formatted_context = LangGraphUtils.format_context(state["context"])
    prompt_input = {
        "question": state["messages"][-1].content,
        "context": formatted_context,
        "chat_history": state["messages"]
    }
    out_prompt = ChatPromptTemplate.from_template(generation_prompt_template_v3).invoke(prompt_input)
    try:
        from langchain_community.callbacks.openai_info import OpenAICallbackHandler
        cb = OpenAICallbackHandler()

        # Initialize the LLM with structured output
        response = await LangGraphConfig.initialize_llm().with_structured_output(
            GeneratedAnswer, method="function_calling"
        ).ainvoke(out_prompt)

        confidence_score = LangGraphUtils.calculate_weighted_confidence_score(response.response_metadata)
        state["answer"] = response.content or "Could not generate an answer."
        state["confidence_score"] = confidence_score

        # Access token usage and cost from the callback handler
        return {
            "answer": state["answer"],
            "context": state["context"],
            "confidence_score": state["confidence_score"],
            "input_tokens": cb.prompt_tokens,
            "output_tokens": cb.completion_tokens,
            "total_tokens": cb.total_tokens,
            "cost": state["cost"] + cb.total_cost
        }
    except Exception:
        logger.error("Error generating answer:\n" + traceback.format_exc())
        return None

This setup allows you to track the token usage and cost using the OpenAICallbackHandler without encountering errors related to structured output ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[AsmaaMHadir]

@dosu I get this error: AttributeError: 'GeneratedAnswer' object has no attribute 'response_metadata'

[dosubot[bot]]

The error AttributeError: 'GeneratedAnswer' object has no attribute 'response_metadata' suggests that the GeneratedAnswer object returned by the LLM call does not have a response_metadata attribute. To resolve this, you should first verify the structure of the GeneratedAnswer class to ensure it includes the necessary attributes.

If response_metadata is expected to be part of the response, you might need to check how the GeneratedAnswer class is defined and ensure it includes this attribute. If response_metadata is not part of the GeneratedAnswer class, you will need to adjust your code to access the correct attributes that are available in the GeneratedAnswer object.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}