would the suggested fix here be to check for non-zero value? @mdonaj

I can try it tomorrow and drop the PR if that works? But seems pragmatic and quick 🥸

That would be great, thank you @mdonaj

have same issue

Seeing a similar issue w/ Deepseek.

I've pushed #10284. I am not sure it solves Deepseek @jbellis. I only tried Anthropic.

@mdonaj it's almost good, but cache_read_input_tokens is included in prompt tokens.
eg.

"cache_read_input_tokens": 5336
"prompt_tokens": 5418

but raw response:
cache_read_input_tokens: 5336
input_tokens: 82

Hmm, I know what you mean. I look at old logs that I have (v1.59.x) and usage coming back from litellm was:

  "usage": {
    "total_tokens": 698,
    "prompt_tokens": 4,
    "completion_tokens": 694,
    "prompt_tokens_details": null,
    "cache_read_input_tokens": 22507,
    "completion_tokens_details": null,
    "cache_creation_input_tokens": 18
  }

and prompt_tokens did not include the cache_read_input_tokens.

But my PR should not have changed this behavior. It must be another thing. Good catch.

edit:

I can see that added in here https://github.com/BerriAI/litellm/blob/0cc0da37f3764bf1db3d07f2369b41844bba8dc3/litellm/llms/anthropic/chat/transformation.py#L638:675

I believe prompt tokens should be the total input tokens - which is then broken down in the prompt_token_details by text_tokens, cached tokens, etc.

This is then used in the token counting logic as well - https://github.com/BerriAI/litellm/blob/main/litellm/litellm_core_utils/llm_cost_calc/utils.py#L159

It makes sense, I do not remember version numbers but in the past it was like that @krrishdholakia and later it was not included and now it is included again.

The period where it was not included (the log that I attached before) seems like a regression. We moved to calculating pricing internally since then. I know my team was recalculating the cost and adjusting for the regression in the past.

We are now we should migrate back to subtracting cache reads and writes from prompt tokens. The pricing on anthropic is different for cache writes, cache reads and input tokens which are all part of prompt_tokens.

Hey @mdonaj

cache writes is not included in prompt tokens - just the cache_read_input_tokens, as cache write is not input to model, just the stuff they stored -

we handle cost tracking for all 3 in generic_cost_per_token (allows it to work for bedrock, vertex, etc.)