crash on long string of repeated "X"s

[biobootloader]

@MentatBot

I use tiktoken as a dependency and discovered a bug.

Script to reproduce:

import tiktoken

bad_string = "X" * 1000000
encoder = tiktoken.get_encoding("cl100k_base")
token_count = len(encoder.encode(bad_string))
print(f"Token count: {token_count}")

Result of running the script:

thread '<unnamed>' panicked at src/lib.rs:250:33:
called `Result::unwrap()` on an `Err` value: RuntimeError(StackOverflow)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "/Users/biobootloader/code/butler/reproduce_crash.py", line 5, in <module>
    token_count = len(encoder.encode(bad_string))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/biobootloader/code/butler/.venv/lib/python3.12/site-packages/tiktoken/core.py", line 124, in encode
    return self._core_bpe.encode(text, allowed_special)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: RuntimeError(StackOverflow)
python reproduce_crash.py  0.16s user 0.03s system 91% cpu 0.197 total

can you figure this out and fix?

[mentatbot]

📦 This agent has been archived and is no longer active.

If you would like to reactivate it, you can unarchive it here.
If you tag mentatbot again a new agent will be created for this issue.

[mentatbot]

Hi @biobootloader! I've implemented a fix for the stack overflow issue you reported. PR #4 is now ready for your review.

The issue

The problem occurs when the byte pair encoding (BPE) algorithm in tiktoken's Rust backend processes very long repetitive strings. The recursion in the algorithm grows too deep and causes a stack overflow.

My fix

I implemented a Python-level solution that:

1. Catches runtime errors containing "StackOverflow" in the message
2. When caught, processes the input text in smaller chunks (10,000 characters each)
3. Combines the results from each chunk

This approach:

• Only activates when needed (preserving normal performance)
• Handles both regular encoding and numpy encoding consistently
• Works with special tokens correctly

Testing

I've added comprehensive tests to verify the fix works, including tests with 100K character strings and tests with special tokens interspersed with repetitive content.

Let me know if you have any questions or would like any adjustments to the implementation!