TokenCountingHandler don't work for OpenAIAgent Function Call

[andreabrgit]

Hi,

The TokenCountingHandler works fine when I do a question to a OpenAIAgent that has a QueryEngineTool inside and use it about indexed documents. But if I do a question to the same OpenAIAgent and it use only the FunctionTools, the TokenCountingHandler don't have any value. How should I use the TokenCountingHandler in this context? Thank you

[dosubot[bot]]

TokenCountingHandler only tracks tokens for LLM and Embedding events that are emitted through the CallbackManager system. When you use QueryEngineTool, it interacts with an LLM and emits these events, so TokenCountingHandler works as expected. However, FunctionTools usually just wrap Python functions, which don't necessarily call an LLM or emit any LLM events—so TokenCountingHandler doesn't see anything to count and remains empty. If your FunctionTool does internally call an LLM (and uses a properly configured CallbackManager), token usage will be tracked; otherwise, it won't be counted by TokenCountingHandler see code reference.

If you need to track token usage for FunctionTools that do invoke LLMs, you can implement custom logic to access the raw OpenAI response and extract token usage, as suggested by maintainers see discussion. For pure Python or non-LLM tools, there simply aren't any tokens to count from the LLM's perspective, so TokenCountingHandler won't show any values.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[dosubot[bot]]

Yes, the prompt and response synthesis should always be counted as token consumption, even if you're only using FunctionTools. The architecture ensures that every user-facing reply from OpenAIAgent is generated via an LLM call, which triggers the CallbackManager and is tracked by TokenCountingHandler as LLM events—so you should see token usage for the prompt and the synthesized response in all cases where the agent replies to the user through the LLM, regardless of which tools were used internally¹².

The only exception is if your FunctionTool does not result in an LLM-generated user response (for example, if you bypass the LLM entirely and return a tool output directly, which is not standard). In normal usage, prompt/response token usage should always be displayed.

Also, note that OpenAIAgent is deprecated; for new projects, FunctionAgent is recommended¹.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

Footnotes

1. OpenAIAgent always synthesizes user-facing responses via an LLM, which triggers token counting ↩ ↩²

2. TokenCountingHandler counts tokens for LLM events emitted through the CallbackManager ↩

[andreabrgit]

@dosu

I understand that in all situations, the function call and synthesis are controlled by LLM. In this case, I don't understand why completion_llm_token_count, total_embedding_token_count and prompt_llm_token_count are completely zero. Because there is a response data. The API key itself shows token consumption.

Regarding FunctionAgent, unfortunately llamaindex is undergoing many changes. A few months ago, when I started my RAG, all the examples were with OpenAIAgent. I will have to find some time to migrate.

[dosubot[bot]]

This usually means the TokenCountingHandler isn't receiving the LLM events it needs to track tokens, even though the OpenAI API shows usage. The most common causes are: the CallbackManager (with TokenCountingHandler attached) isn't properly connected to both your LLM and the agent, or the LLM response structure has changed and no longer includes the expected 'usage' fields that TokenCountingHandler reads from. There was a recent bug fix for this kind of mismatch—so make sure you're on the latest version if possible (see PR #19126).

To troubleshoot: double-check that when you create your OpenAIAgent, you pass the same CallbackManager (with TokenCountingHandler) to both the agent and the LLM instance. You can also enable verbose logging to confirm that CallbackManager events are firing during LLM calls. If you still see zeros, inspect the raw LLM response to verify it contains a 'usage' or 'usage_metadata' field—TokenCountingHandler depends on this to extract token counts (see code reference).

If none of this resolves it, you can implement a custom solution to extract token usage directly from the raw OpenAI response, as suggested by maintainers (see discussion).

OpenAIAgent is deprecated and not maintained; FunctionAgent is the recommended path going forward (see deprecation notice), but the above steps should help you get accurate token counts in your current setup.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[andreabrgit]

@dosu

I pass the TokenCountingHandler thought settings

    self._callback_manager = CallbackManager(handlers=[self._llama_debug, self._token_counter])
    
    Settings.llm = self._llm
    Settings.embed_model = self._embeddings
    Settings.callback_manager = self._callback_manager

[dosubot[bot]]

The most likely reason TokenCountingHandler isn't tracking tokens is that setting Settings.llm and Settings.callback_manager after creating your LLM or agent does not retroactively update their internal references—so LLM events may not be emitted as expected. For TokenCountingHandler to work, the CallbackManager (with TokenCountingHandler) must be attached directly to the LLM instance before you pass it to the agent, and the same CallbackManager should be passed to the agent itself. Just setting Settings.callback_manager isn't enough if objects are already instantiated see code reference.

To fix this, make sure you do the following before creating your agent:

1. Set the CallbackManager (with TokenCountingHandler) on your LLM:
   self._llm.callback_manager = self._callback_manager
2. Pass the same CallbackManager to the agent constructor (if it accepts it).
3. Only then assign to Settings.llm and Settings.callback_manager if you need global defaults.

Also, recent versions have moved instrumentation (including CallbackManager and TokenCountingHandler) into a standalone package (llama-index-instrumentation), so make sure it's installed and your imports are up to date see PR.

If you still see zeros, double-check that your FunctionTool actually triggers an LLM call for the user-facing response—pure Python tools won't emit LLM events, so no tokens will be counted in those cases.

If you want to future-proof your setup, consider migrating to FunctionAgent and the new instrumentation system, as OpenAIAgent is deprecated.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}