Skip to content

Releases: huggingface/tokenizers

Rust v0.11.2

28 Feb 10:53
Compare
Choose a tag to compare
  • [#919] Fixing single_word AddedToken. (regression from 0.11.2)
  • [#916] Deserializing faster added_tokens by loading them in batch.

Python v0.11.6

28 Feb 09:22
Compare
Choose a tag to compare
  • [#919] Fixing single_word AddedToken. (regression from 0.11.2)
  • [#916] Deserializing faster added_tokens by loading them in batch.

Node v0.8.3

28 Feb 10:34
Compare
Choose a tag to compare
node-v0.8.3

Adding rust release CI.

Python v0.11.5

16 Feb 12:08
Compare
Choose a tag to compare

[#895] Add wheel support for Python 3.10

Rust v0.11.1

17 Jan 09:00
Compare
Choose a tag to compare
  • [#882] Fixing Punctuation deserialize without argument.
  • [#868] Fixing missing direction in TruncationParams
  • [#860] Adding TruncationSide to TruncationParams

Python v0.11.3

17 Jan 09:30
Compare
Choose a tag to compare
  • [#882] Fixing Punctuation deserialize without argument.
  • [#868] Fixing missing direction in TruncationParams
  • [#860] Adding TruncationSide to TruncationParams

Node v0.8.2

17 Jan 21:33
Compare
Choose a tag to compare

[#884] Fixing bad deserialization following inclusion of a default for Punctuation

Node v0.8.1

17 Jan 08:59
Compare
Choose a tag to compare

Fixing various backward compatibility bugs (Old serialized files couldn't be deserialized anymore.

Python v0.11.4

17 Jan 21:32
Compare
Choose a tag to compare
Python v0.11.4 Pre-release
Pre-release

[#884] Fixing bad deserialization following inclusion of a default for Punctuation

Python v0.11.2

04 Jan 13:59
Compare
Choose a tag to compare
Python v0.11.2 Pre-release
Pre-release

Fixes #868