fix: Anthropic prompt caching on GCP Vertex AI #9605
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
krrishdholakia
added a commit
that referenced
this pull request
Apr 1, 2025
This reverts commit a867324.
krrishdholakia
added a commit
that referenced
this pull request
Apr 1, 2025
krrishdholakia
added a commit
that referenced
this pull request
Apr 1, 2025
krrishdholakia
added a commit
that referenced
this pull request
Apr 1, 2025
…x_parallel_requests = 0 (#9671) * fix(proxy_server.py): remove non-functional parent backoff/retry on /chat/completion Causes circular reference error * fix(http_parsing_utils.py): safely return parsed body - don't allow mutation of cached request body by client functions Root cause fix for circular reference error * Revert "fix: Anthropic prompt caching on GCP Vertex AI (#9605)" (#9670) This reverts commit a867324. * add type hints for AnthropicMessagesResponse * define types for response form AnthropicMessagesResponse * fix response typing * allow using litellm.messages.acreate and litellm.messages.create * fix anthropic_messages implementation * add clear type hints to litellm.messages.create functions * fix anthropic_messages * working anthropic API tests * fixes - anthropic messages interface * use new anthropic interface * fix code quality check * docs anthropic messages endpoint * add namespace_packages = True to mypy * fix mypy lint errors * docs anthropic messages interface * test: fix unit test * test(test_http_parsing_utils.py): update tests --------- Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Title
Fix (hopefully) for prompt caching not working with Anthropic models on GCP VertexAI
Relevant issues
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit)[https://docs.litellm.ai/docs/extras/contributing_code] - the same tests fail on my branch as fail on mainNew test:
Other tests are no more broken than they are on main at present:
And the same linting errors as on main (not on the changes included here):
Type
🐛 Bug Fix
Changes
Add missing header for Anthropic models on GCP Vertex which allows prompt caching to work.