Skip to content

Commit 55c8d83

Browse files
authored
added two new embedding model's encoding (#247)
Library doesn't support two new embedding model's encoding mapper - `text-embedding-3-small` - `text-embedding-3-large` Added Encoding mapper for 2 new embedding models. The source of mapping is taken from https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
1 parent 6cc3a46 commit 55c8d83

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

tiktoken/model.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,8 @@
2626
"babbage-002": "cl100k_base",
2727
# embeddings
2828
"text-embedding-ada-002": "cl100k_base",
29+
"text-embedding-3-small": "cl100k_base",
30+
"text-embedding-3-large": "cl100k_base",
2931
# DEPRECATED MODELS
3032
# text (DEPRECATED)
3133
"text-davinci-003": "p50k_base",

0 commit comments

Comments
 (0)