Code

Support "fuse unknown" configuration#117

Code

Merged

pcuenca merged 11 commits intomainhuggingface/swift-transformers:mainfrom

support-fuse-unknownhuggingface/swift-transformers:support-fuse-unknownCopy head branch name to clipboard

Aug 19, 2024

Commits

Commits on Aug 2, 2024

• Bring over hf token envvar from preview branch

  pcuenca

  committed
  01c0f25

  Copy full SHA for 01c0f25

• Add tests for Gemma, including edge cases

  Show description for 4149b55
  pcuenca

  committed
  4149b55

  Copy full SHA for 4149b55

• Sort added tokens by length (descending) to avoid early partial matches

  Show description for 59a5b3a
  pcuenca

  committed
  59a5b3a

  Copy full SHA for 59a5b3a

• Store vocab as NSString to allow multiple tokens with the same Unicode

  Show description for 4631505
  pcuenca

  committed
  4631505

  Copy full SHA for 4631505

• Remove comments

  pcuenca

  authored
  a6637e0

  Copy full SHA for a6637e0

• Go back to making vocab dictionaries private

  pcuenca

  committed
  266bbd0

  Copy full SHA for 266bbd0

• Use ungated copy of Gemma tokenizer

  pcuenca

  committed
  f2044bd

  Copy full SHA for f2044bd

Commits on Aug 3, 2024

• Use NSString in UnigramTokenizer

  pcuenca

  committed
  b610c2d

  Copy full SHA for b610c2d

• Support fuse_unk configuration.

  Show description for af8483b
  pcuenca

  committed
  af8483b

  Copy full SHA for af8483b

Commits on Aug 19, 2024

• Merge remote-tracking branch 'origin/main' into support-fuse-unknown

  pcuenca

  committed
  767e8e6

  Copy full SHA for 767e8e6

• Remove duplicate function

  pcuenca

  committed
  290cc0a

  Copy full SHA for 290cc0a