track token usage for each tool call in ChatAgent #3325
Conversation
Add token_usage field to ToolCallingRecord to monitor the cost of each individual tool call. This enables fine-grained cost tracking when multiple tools are invoked in a single agent step. Changes: - Add token_usage field to ToolCallingRecord - Update ChatAgent to pass token usage from LLM responses to tool records - Add unit test for token tracking - Add example demonstrating the feature
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the ✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
@LIHUA919 thanks for your contribution？ left some comment below
There was a problem hiding this comment.
try to test this example ，occur error
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/suntao/Documents/GitHub/camel/camel/agents/chat_agent.py", line 3121, in _record_tool_calling
tool_record = ToolCallingRecord(
^^^^^^^^^^^^^^^^^^
File "/Users/suntao/Documents/GitHub/camel/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 2 validation errors for ToolCallingRecord
token_usage.completion_tokens_details
Input should be a valid integer [type=int_type, input_value={'accepted_prediction_tok...d_prediction_tokens': 0}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.12/v/int_type
token_usage.prompt_tokens_details
Input should be a valid integer [type=int_type, input_value={'audio_tokens': 0, 'cached_tokens': 0}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.12/v/int_type| # Process all tool calls | ||
| # All tools in this batch share the same token usage from the | ||
| # LLM call that generated them | ||
| current_token_usage = response.usage_dict |
There was a problem hiding this comment.
i think after multiturn request response.usage_dict cannot accurately represent the usage of a toolcall
4 participants
Summary
This PR implements fine-grained token usage tracking for individual tool calls in ChatAgent, addressing issue #3219.
Changes
token_usagefield toToolCallingRecordto store token usage information for each tool callChatAgent._record_tool_calling()to accept and store token usage dataChatAgent._execute_tool()and_aexecute_tool()to propagate token usage from LLM responses_step_impl()and_astep_impl()to pass token usage to tool recordsToolCallingRecord.__str__()to display token usage when availableTechnical Details
When multiple tools are called in a single LLM response, they share the same token usage dict from that response. This allows users to monitor the cost of each tool call and
optimize agent workflows for cost efficiency.
The
token_usagedict contains standard OpenAI-format keys:prompt_tokens: Number of tokens in the promptcompletion_tokens: Number of tokens in the completiontotal_tokens: Total tokens usedUse Case
This feature is particularly valuable for cost-sensitive applications like the Eigent search agent mentioned in the issue, where different tools (e.g., Google Search vs browser
tools) have varying token costs.
Testing
test_tool_calling_token_usage_trackingexamples/agents/tool_calling_with_token_tracking.py