We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
There was an error while loading. Please reload this page.
1 parent 1f098ca commit ab3688aCopy full SHA for ab3688a
README.md
@@ -1,11 +1,12 @@
1
# ⏳ tiktoken
2
3
-tiktoken is a fast tokeniser.
+tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with
4
+OpenAI's models.
5
6
```python
7
import tiktoken
8
enc = tiktoken.get_encoding("gpt2")
-print(enc.encode("hello world"))
9
+assert enc.decode(enc.encode("hello world")) == "hello world"
10
```
11
12
The open source version of `tiktoken` can be installed from PyPI:
0 commit comments