Title

Fix UnicodeDecodeError on non-English OS by specifying UTF-8 encoding when opening files

Relevant issues

Fixes #10264

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added testing in the tests/litellm_utils_tests/test_utils.py directory, Adding at least 1 test is a hard requirement – see details at https://docs.litellm.ai/docs/extras/contributing_code
• I have added a screenshot/log of my new test passing locally
• My PR passes all unit tests on (make test-unit)
• My PR’s scope is as isolated as possible; it only solves 1 specific problem

Type

🐛 Bug Fix

Changes

• Changed all occurrences of open("r") to open("r", encoding="utf-8") to ensure files are read as UTF-8 rather than the platform default, preventing UnicodeDecodeError on non-English OSes.
• Added a unit test at tests/litellm_utils_tests/test_utils.py to verify that anthropic_tokenizer.json is read correctly with UTF-8 encoding.

screenshot