Skip to content

Commit ab3688a

Browse files
committed
README.md: minor improvements
1 parent 1f098ca commit ab3688a

File tree

1 file changed

+3
-2
lines changed

1 file changed

+3
-2
lines changed

README.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
11
# ⏳ tiktoken
22

3-
tiktoken is a fast tokeniser.
3+
tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with
4+
OpenAI's models.
45

56
```python
67
import tiktoken
78
enc = tiktoken.get_encoding("gpt2")
8-
print(enc.encode("hello world"))
9+
assert enc.decode(enc.encode("hello world")) == "hello world"
910
```
1011

1112
The open source version of `tiktoken` can be installed from PyPI:

0 commit comments

Comments
 (0)