We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
There was an error while loading. Please reload this page.
1 parent ab3688a commit 4226a6cCopy full SHA for 4226a6c
tiktoken/core.py
@@ -59,7 +59,7 @@ def encode(
59
) -> list[int]:
60
"""Encodes a string into tokens.
61
62
- Special tokens are tokens are artificial tokens used to unlock capabilities from a model,
+ Special tokens are artificial tokens used to unlock capabilities from a model,
63
such as fill-in-the-middle. So we want to be careful about accidentally encoding special
64
tokens, since they can be used to trick a model into doing something we don't want it to do.
65
0 commit comments