How to get token cost from a langGraph based implemented Openai Model

[FawazSapa]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_core.messages import HumanMessage

inputs = {
    "messages": [
        HumanMessage(
            content='Can you delete the user with email mike@gmail.com.'
        )
    ]
}

# Use the Runnable to get the final state
final_state = app.invoke(
    inputs,
    config={"configurable": {"thread_id": 42}}
)

# Print the final response
print(final_state)

Description

I have developed a tool utilizing OpenAI’s GPT-3.5-turbo-1106, integrated with LangSmith and LangGraph. I am looking to implement a feature that limits the usage of my bot based on a certain cost threshold for OpenAI API calls.
The final_state doesn’t provide any information regarding tokens, unlike the documentation that used the metadata from the response.
I tried using get_openai_callback function but it didnt work either.
How do I get to know the tokens usage per api?

System Info

langchain==0.2.11
langchain-community==0.2.10
langchain-core==0.2.23
langchain-openai==0.1.17
langchain-text-splitters==0.2.2

[dosubot[bot]]

Hello @FawazSapa! I'm here to help you with your GitHub issue. If you need assistance, just let me know!

To get token usage and cost information from a LangGraph-based implementation of an OpenAI model, you can use the OpenAICallbackHandler class. Here is an example of how to integrate it into your code:

1. Initialize the OpenAICallbackHandler:
2. Pass the callback handler to your model invocation:
3. Access the token usage information from the callback handler:

Here is the updated code:

from langchain_core.messages import HumanMessage
from langchain_community.callbacks.openai_info import OpenAICallbackHandler

# Initialize the callback handler
callback_handler = OpenAICallbackHandler()

inputs = {
    "messages": [
        HumanMessage(
            content='Can you delete the user with email mike@gmail.com.'
        )
    ]
}

# Use the Runnable to get the final state with the callback handler
final_state = app.invoke(
    inputs,
    config={"configurable": {"thread_id": 42}},
    callbacks=[callback_handler]  # Pass the callback handler here
)

# Print the final response
print(final_state)

# Access and print the token usage information
print(f"Total Tokens Used: {callback_handler.total_tokens}")
print(f"Prompt Tokens: {callback_handler.prompt_tokens}")
print(f"Completion Tokens: {callback_handler.completion_tokens}")
print(f"Successful Requests: {callback_handler.successful_requests}")
print(f"Total Cost (USD): ${callback_handler.total_cost}")

This approach ensures that the OpenAICallbackHandler tracks the token usage and cost information during the model invocation. You can then access this information from the callback handler after the invocation is complete ^[1][2].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

To address the issues you're encountering, let's break down the solutions:

1. Accessing usage_metadata:
   The error AttributeError: 'AddableValuesDict' object has no attribute 'usage_metadata' indicates that the final_state object does not have the usage_metadata attribute. This suggests that the response object from app.invoke is not directly providing the token usage information.

2. Using get_openai_callback:
   If the get_openai_callback context manager is returning 0 total tokens, it might be due to the way the callback is being integrated or the specific model/version being used.

Solution 1: Custom Callback Handler

You can create a custom callback handler to track token usage and integrate it with your LangGraph-based implementation. Here is an example:

from langchain_core.messages import HumanMessage
from langchain_community.callbacks.openai_info import OpenAICallbackHandler

# Initialize the callback handler
callback_handler = OpenAICallbackHandler()

inputs = {
    "messages": [
        HumanMessage(
            content='Can you delete the user with email mike@gmail.com.'
        )
    ]
}

# Use the Runnable to get the final state with the callback handler
final_state = app.invoke(
    inputs,
    config={"configurable": {"thread_id": 42}},
    callbacks=[callback_handler]  # Pass the callback handler here
)

# Print the final response
print(final_state)

# Access and print the token usage information
print(f"Total Tokens Used: {callback_handler.total_tokens}")
print(f"Prompt Tokens: {callback_handler.prompt_tokens}")
print(f"Completion Tokens: {callback_handler.completion_tokens}")
print(f"Successful Requests: {callback_handler.successful_requests}")
print(f"Total Cost (USD): ${callback_handler.total_cost}")

Solution 2: Using get_openai_callback Correctly

Ensure that the get_openai_callback context manager is correctly tracking the token usage. Here is an example:

from langchain_community.callbacks import get_openai_callback
from langchain_openai import OpenAI

llm = OpenAI(model_name="gpt-3.5-turbo-instruct")

with get_openai_callback() as cb:
    result = llm.invoke("Tell me a joke")
    result2 = llm.invoke("Tell me a joke")
    print(f"Total Tokens: {cb.total_tokens}")
    print(f"Prompt Tokens: {cb.prompt_tokens}")
    print(f"Completion Tokens: {cb.completion_tokens}")
    print(f"Total Cost (USD): ${cb.total_cost}")

Additional Debugging

If the above solutions do not resolve the issue, consider the following debugging steps:

1. Check Model Version:
   Ensure that the model version you are using supports token usage tracking. Some versions might not provide this information.

2. Verify Callback Integration:
   Double-check that the callback handler is correctly integrated and that the model invocation is correctly passing the callback.

3. LangSmith Integration:
   If you are using LangSmith, ensure that the token usage information is being correctly logged and accessible through the LangSmith API or interface.

By following these steps, you should be able to track and access token usage information in your LangGraph-based implementation of an OpenAI model ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[EarningsCall]

@dosu how do I calculate costs for anthropic models?

[wexoliver]

To whoever tries the code in Solution 1 above, I'd like to note that I had success putting the callbacks into the config like this:

invocation_result = graph.invoke(
    inputs,
    config={
        "configurable": {"thread_id": 42},
        "callbacks": [callback_handler],
    },
)

[mrctito]

@wexoliver , thank you shareing with us!

Maybe you could help me understand what I am still doing wrong.

All these values are always zeroed out:

print(f"Total Tokens Used: {callback_handler.total_tokens}")
print(f"Prompt Tokens: {callback_handler.prompt_tokens}")
print(f"Completion Tokens: {callback_handler.completion_tokens}")
print(f"Successful Requests: {callback_handler.successful_requests}")
print(f"Total Cost (USD): ${callback_handler.total_cost}")

Thank you!

[wexoliver]

Hey @mrctito -- you might have to add your model name to the MODEL_COST_PER_1k_TOKENS dictionary (source here). Depending on your implementation, you might have to make a copy of this callback in your source code, or ensure you are instantiating the callback with the correct values.

[binstitus]

response = await graph.ainvoke(state, config,callbacks=[callback_handler])

{
"detail": "Pregel.astream() got an unexpected keyword argument 'callbacks'"
}

[wexoliver]

hey @binstitus -- you are configuring your callbacks incorrectly. config is a dictionary with a key for callbacks that takes on a list of callbacks. Also, since you are using ainvoke you need to ensure that your callback implements the AsyncCallbackHandler, or else you are going to run into threading / async errors (source here)