Utility to calculate LLM context length of an arbitrary string.
Based on https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them.
str_to_tokens(str: str) -> floatstr_to_processing_tokens(str: str) -> int-
Notifications
You must be signed in to change notification settings - Fork 0
surfly/llm-context-length
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
No description, website, or topics provided.
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published