Skip to content

Commit 4226a6c

Browse files
authored
Remove duplicate words in docstring (#3)
1 parent ab3688a commit 4226a6c

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

tiktoken/core.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ def encode(
5959
) -> list[int]:
6060
"""Encodes a string into tokens.
6161
62-
Special tokens are tokens are artificial tokens used to unlock capabilities from a model,
62+
Special tokens are artificial tokens used to unlock capabilities from a model,
6363
such as fill-in-the-middle. So we want to be careful about accidentally encoding special
6464
tokens, since they can be used to trick a model into doing something we don't want it to do.
6565

0 commit comments

Comments
 (0)