Fix issue 10264 unicode docode error#10350

Closed

sugarkwork wants to merge 2 commits intoBerriAI:mainfrom

sugarkwork:fix-issue-10264-unicode-docode-error

Commits on Apr 27, 2025

Specify UTF-8 encoding when opening files to avoid cp932 decode errors
sugarkwork
committed
add test for open(..., encoding="utf-8") in anthropic_tokenizer.json (fixes #10264)
sugarkwork
committed