Getting error "Inconsistent number of segments: whisper_segments ({l1}"

[amitli1]

Sometimes I'm getting error:

line 870, in _transcribe_timestamped_efficient
    assert l1 == l2 or l1 == 0, f"Inconsistent number of segments: whisper_segments ({l1}) != timestamped_word_segments ({l2})"
AssertionError: Inconsistent number of segments: whisper_segments (1) != timestamped_word_segments (0)

when using whisper.transcribe.

I'm running with the following code:

whisper.transcribe(
                model,
                audio,             
                initial_prompt=buffer, # compilation of all transcript history
                verbose=True  
            )

What is wrong ?

[Jeronymous]

Thanks for reporting. Too bad to have this error again.

I would need more details to be able to reproduce and investigate:
some audio (it can be a pickled torch array, if not a file), the details of the model that is used, as well as the initial_prompt.

[dominikmau]

@Jeronymous
Hello, thanks for the great project.
Unfortunately I get the same error:

Inconsistent number of segments: whisper_segments (1) != timestamped_word_segments (2)
Traceback (most recent call last):
  File "C:\Users\test\Desktop\whisper2\whisper_test.py", line 7, in <module>
    result = whisper.transcribe(model, audio, language="de")
  File "C:\Users\test\Desktop\whisper2\venv\lib\site-packages\whisper_timestamped\transcribe.py", line 296, in transcribe_timestamped
    (transcription, words) = _transcribe_timestamped_efficient(model, audio,
  File "C:\Users\test\Desktop\whisper2\venv\lib\site-packages\whisper_timestamped\transcribe.py", line 920, in _transcribe_timestamped_efficient
    assert l1 == l2 or l1 == 0, f"Inconsistent number of segments: whisper_segments ({l1}) != timestamped_word_segments ({l2})"
AssertionError: Inconsistent number of segments: whisper_segments (1) != timestamped_word_segments (2))

Happens to me with every audio file, here is an example - test.mp3, 00:00:56:
https://mega.nz/file/EWpCiD5b#FqW20Gy-StHcd9dskoOe3pEPuFxSJyzfOUhk6kd4NmQ

I use a finetuned model:
https://huggingface.co/primeline/whisper-large-v3-german

Python version:

Python 3.10.11

Installed Python packages:

certifi==2024.7.4
charset-normalizer==3.3.2
colorama==0.4.6
Cython==3.0.10
dtw-python==1.5.1
filelock==3.13.1
fsspec==2024.2.0
huggingface-hub==0.23.4
idna==3.7
intel-openmp==2021.4.0
Jinja2==3.1.3
llvmlite==0.43.0
MarkupSafe==2.1.5
mkl==2021.4.0
more-itertools==10.3.0
mpmath==1.3.0
networkx==3.2.1
numba==0.60.0
numpy==1.26.3
openai-whisper==20231117
packaging==24.1
pillow==10.2.0
PyYAML==6.0.1
regex==2024.5.15
requests==2.32.3
safetensors==0.4.3
scipy==1.14.0
sympy==1.12
tbb==2021.11.0
tiktoken==0.7.0
tokenizers==0.19.1
torch==2.3.1+cu118
torchaudio==2.3.1+cu118
torchvision==0.18.1+cu118
tqdm==4.66.4
transformers==4.42.3
typing_extensions==4.9.0
urllib3==2.2.2
whisper-timestamped==1.15.4

My python code:

import whisper_timestamped as whisper

audio = whisper.load_audio("test.mp3")
model = whisper.load_model("model", device="cuda")
result = whisper.transcribe(model, audio, language="de")

import json
with open('data.json', 'w', encoding='utf-8') as f:
    json.dump(result, f, ensure_ascii=False, indent=4)

Debug:

(venv) C:\Users\test\Desktop\whisper2>python whisper_test.py
100%|██████████████████████████████████████████████████████████████████████████| 5600/5600 [03:12<00:00, 29.03frames/s]

(venv) C:\Users\test\Desktop\whisper2>python whisper_test.py
 54%|███████████████████████████████████████                                  | 3000/5600 [00:14<00:12, 200.77frames/s]DEBUG:whisper_timestamped:Add segment 1 at step 114:
        <|transcribe|><|0.00|>Also gut, sagte die Denkmaschine und erhob sich. Sagen wir, die Todeszelle im Chisholm-Gefängnis. Die Todeszelle im Chisholm-Gefängnis. Und was werden sie anziehen? So wenig wie möglich, sagte die Denkmaschine. Schuhe, Strümpfe, Hose und ein Hemd. Sie werden sich natürlich durchsuchen lassen. Ich werde exakt so behandelt, wie alle Gefangenen behandelt werden.
DEBUG:whisper_timestamped:WARNING: end timestamp not produced. Adding <|endoftext|>
DEBUG:whisper_timestamped:Missing end token in <|0.00|>Also gut, sagte die Denkmaschine und erhob sich. Sagen wir, die Todeszelle im Chisholm-Gefängnis. Die Todeszelle im Chisholm-Gefängnis. Und was werden sie anziehen? So wenig wie möglich, sagte die Denkmaschine. Schuhe, Strümpfe, Hose und ein Hemd. Sie werden sich natürlich durchsuchen lassen. Ich werde exakt so behandelt, wie alle Gefangenen behandelt werden.<|endoftext|>
100%|█████████████████████████████████████████████████████████████████████████| 5600/5600 [00:44<00:00, 126.38frames/s]
DEBUG:whisper_timestamped:Add segment 2 at step 337:
        <|transcribe|><|0.00|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|>
DEBUG:whisper_timestamped:WARNING: decoding hit the max limit for segment [50360, 50365, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261, 50261] (It usually happens when the language model gets stuck)
DEBUG:whisper_timestamped:         Guess last token using probas (assuming greedy decoding): 50261
DEBUG:whisper_timestamped:WARNING: also add last token: <|de|>
DEBUG:whisper_timestamped:Missing end token in <|0.00|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|><|de|>
WARNING:whisper_timestamped:Inconsistent number of segments: whisper_segments (1) != timestamped_word_segments (2)
Traceback (most recent call last):
  File "C:\Users\test\Desktop\whisper2\whisper_test.py", line 12, in <module>
    result = whisper.transcribe(model, audio, language="de")
  File "C:\Users\test\Desktop\whisper2\venv\lib\site-packages\whisper_timestamped\transcribe.py", line 296, in transcribe_timestamped
    (transcription, words) = _transcribe_timestamped_efficient(model, audio,
  File "C:\Users\test\Desktop\whisper2\venv\lib\site-packages\whisper_timestamped\transcribe.py", line 920, in _transcribe_timestamped_efficient
    assert l1 == l2 or l1 == 0, f"Inconsistent number of segments: whisper_segments ({l1}) != timestamped_word_segments ({l2})"
AssertionError: Inconsistent number of segments: whisper_segments (1) != timestamped_word_segments (2)

As mentioned in other issues:

naive_approach = True

To make it work at all, but then only a part of the audio is transcribed.

[Jeronymous]

Thanks a lot @dominikmau
Somehow I missed this message during the summer, but what you give is super useful to reproduce and understand/solve the issue (thanks for the professional issue reporting 🙏).

Only problem : the link to the audio is broken. Probably because I come here too late... Can you please resend (if it's a small file, it can be enclosed here, by zipping and drag&dropping the zip file in this discussion)

[yurkoturskiy]

Hi @Jeronymous

I experience the same Inconsistent number of segments issue. I have a lot of assets and happy to share them privately. Can I use your linagora.com email to send the link?

[Jeronymous]

Yes @yurkoturskiy ! Thanks
A few examples to reproduce should be enough.
The shorter the better :)

[yurkoturskiy]

Great! I've made a folder and shared the link over email. Please, let me know if there is anything I can do to help to replicate and fix it.
Thank you!

[ahmedHanzala]

Did we manage to find a fix for this?

[hwscut]

anyone fix this bug? also get same error

[SLbaba]

I am encountering a similar issue when running the batch ['fe_03_10495', 'fe_03_08817', 'fe_03_06312', 'fe_03_01476'](Fisher Eval Datasets). The error message is as follows:

AssertionError: Inconsistent number of segments: whisper_segments (323) != timestamped_word_segments (321)
assert l1 == l2 or l1 == 0, f"Inconsistent number of segments: whisper_segments ({l1}) != timestamped_word_segments ({l2})"

[Jeronymous]

@hwscut @SLbaba what version of openai-whisper do you have ?

pip index versions openai-whisper

I recently saw cases that were failing with openai-whisper==20230314 which were all solved by using the latest version of openai-whisper

pip install openai-whisper==20240930

So I am wondering if upgrading whisper like this would not be the solution to this annoying bug.

(it would not be the case if one of you observe the failure with the latest version 20240930)

[yurkoturskiy]

I'll evaluate the latest whisper, but as you said, it might not be a solution. I'll let you know how it goes.

[SLbaba]

[Jeronymous]

Thank you @SLbaba
can you please give the exact thing to reproduce (all options of command line or python transcribe() function)?

['fe_03_10495', 'fe_03_08817', 'fe_03_06312', 'fe_03_01476'](Fisher Eval Datasets).

Can you also please indicate where to find those audios ?

[chenkigba]

@hwscut @SLbaba what version of openai-whisper do you have ?您有什么版本的 OpenAI-whisper ？

pip index versions openai-whisper

I recently saw cases that were failing with openai-whisper==20230314 which were all solved by using the latest version of openai-whisper我最近看到了一些使用 openai-whisper==20230314 失败的情况，这些情况都是通过使用最新版本的 openai-whisper 来解决的

pip install openai-whisper==20240930

So I am wondering if upgrading whisper like this would not be the solution to this annoying bug.所以我想知道像这样升级 whisper 是否不是这个烦人错误的解决方案。

(it would not be the case if one of you observe the failure with the latest version 20240930)（如果你们中的一个人观察到最新版本 20240930 的失败，则不会出现这种情况）

Some Info might help:

Same audio file. Same other params (I use vad with (start, end) list). Except:

suppress_tokens

SUPPRESS_TOKENS = [
    [50365, 11914, 5233, 235, 220, 31206, 2415, 245, 5419, 249, 50415],
    [50465, 38109, 244, 30753, 220, 31206, 2415, 245, 5419, 249, 50515],
    [50515, 25491, 11914, 4035, 220, 31206, 2415, 245, 5419, 249, 50565],
    [50365, 27908, 1960, 2392, 251, 12579, 5266, 252, 220, 7422, 95, 10034, 227, 220, 17819, 105, 28926,
        220, 12467, 5266, 237, 35488, 11100, 12373, 250, 940, 236, 12579, 12579, 8987, 237, 11386, 51864]
]
SUPPRESS_TOKENS = [token for sublist in SUPPRESS_TOKENS for token in sublist] 

suppress_tokens=SUPPRESS_TOKENS if kwargs.get("use_supress_tokens", False) else None,

When suppress_tokens is None, everything ok, except in my this case, it will give some repeated, covered, wrong results like:

['累积所有对你的思念', '把思念推进个黑暗的房间', '房间就在厨房的后面', '没人会发现', '言不离手把自己关醉', '醉醒过后笑自己有多狼狈', '泡的咖啡有苦的滋味', '我张开了忘记你是谁', '让风筝自由干脆剪断了线', '让它望蓝天空随风而飞', '越飞越好直到看不见', '像电影里的画面', '这样也许能让我好过一些', '把你当作风筝随风而飞', '越飞越好直到无所谓', '我剪断了线', '不再对你怀念', '嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯', '嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯嗯']

Which '嗯嗯嗯嗯嗯嗯...' is accually start with some '嗯' and later other words not '嗯'.NOTICE: '嗯' word is singing by humming. It's a BUG.

As I mentioned, I use vad with (start, end) list. This picture shows the vad segments.

Later, when I give suppress tokens, error like AssertionError: Inconsistent number of segments: whisper_segments (89) != timestamped_word_segments (88) happened.

[chenkigba]

According to Gemini-2.5-pro's analysis, it might related with the difference between your hook and whisper native process with suppress tokens.

It recommend me to use navtive_approch=True, and it works to avoid AssertionError: Inconsistent number of segments: whisper_segments (89) != timestamped_word_segments (88)

Though it's still full of error.( In a different way)

['堆积所有对你的思念', '把思念推进个黑暂的房间', '房间就在厨房的后面', '没任何会发现', '言不了手把自己关醉', '醉醒过后笑自己有多狼狈', '泡的咔啡有苦的滋味', '我张开了忘记你是谁', '让风筹自由干脆剪断了线', '让它望蓝天空隋风', '像电影里的画面', '这样也许能让我好过一些', '把你当做风筹', '隋风', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线', '我剪断了线']

Gemini-2.5-pro's raw reply:

---

This is a very insightful question that gets to the heart of how whisper-timestamped's efficient mode works and where its vulnerabilities lie.

When you pass the suppress_tokens parameter, the inconsistency between whisper_segments (from the official Whisper output) and timestamped_word_segments (from the library's simulation) arises from a fundamental conflict: suppress_tokens directly interferes with the core decoding process inside the Whisper model, whereas the library's efficient mode is merely an external "imitation" that observes and simulates this process using hooks. When this imitation fails to perfectly replicate all the side effects of the suppress_tokens interference, the two results diverge.

Let's break down the specific reasons for this inconsistency:

1. Suppression of Critical Timestamp Tokens (The Primary Cause)

The core logic for segmenting audio in the efficient mode (the must_flush_segment function) relies on detecting consecutive timestamp tokens. For instance, when the model generates a sequence like <|0.52|><|...|> ... text ... <|5.24|><|6.01|>, the library identifies the end of one segment and the beginning of the next between <|5.24|> and <|6.01|>.

If the list of token IDs you provide in suppress_tokens includes any of the timestamp tokens, it leads to catastrophic failure:

• The whisper-timestamped library's perspective: Its hooks observe that the decoder never generates the suppressed timestamp token. As a result, it may never receive the "consecutive timestamp" signal it needs to correctly split the segments. It might merge multiple segments that should have been separate into a single one, or fail to create any segments at all.
• The internal Whisper model's perspective: Whisper's native transcribe function has more robust internal logic to handle this. For example, if the highest-probability token is a suppressed timestamp, it might choose the next-best token or simply force-end the current segment when it reaches the 30-second window boundary.

The result: The internal Whisper model proceeds to generate N segments (whisper_segments), but the library's simulator, having missed its crucial signal, only generates M segments (timestamped_word_segments), where M != N. This directly causes the assert l1 == l2 check to fail.

2. Interference with the Decoder's Logit Filters

The suppress_tokens parameter is implemented at a low level via a mechanism called a LogitFilter. At each step of the decoding process, this filter sets the probability (logit) of the specified tokens to negative infinity, making them impossible to select.

The problem is that in efficient mode, two separate entities are creating and using these LogitFilters:

1. The Whisper transcribe function: It creates its own set of LogitFilters for its internal decoding loop.
2. The whisper-timestamped get_logit_filters function: It creates its own set of LogitFilters to simulate the decoding process.

Although both aim to implement suppress_tokens, their implementation details or their interaction with other filters might differ slightly. For instance, Whisper might have hard-coded rules (e.g., "never suppress the <|endoftext|> token under any circumstances") that the library's simulation doesn't account for. This minor logical difference can be amplified during a complex decoding process, causing their final token sequences to diverge and, consequently, affecting the segmentation.

3. Disruption of Decoding and Fallback Mechanisms

Whisper employs a complex set of mechanisms to handle decoding failures (like generating gibberish or getting stuck in a repetitive loop), which involves checking thresholds like logprob_threshold and compression_ratio_threshold. When certain conditions are met, it will "fallback" by, for example, increasing the sampling temperature and re-decoding the segment.

When you use suppress_tokens, you might inadvertently prevent the model from generating a normal, high-probability token, forcing it to choose a low-probability one that leads to a decoding failure. In this scenario:

• Inside Whisper: This might trigger its internal fallback mechanism, which ultimately succeeds in generating valid segments using a different set of parameters.
• In the whisper-timestamped simulator: While it also observes metrics like logprob, its simulation of the fallback mechanism is likely incomplete. It cannot predict that Whisper will "start over" with a new attempt, causing the decoding history it recorded to be inconsistent with the "correct history" that Whisper finally adopted.

Summary and Analogy

You can think of whisper-timestamped's efficient mode as a live TV commentator who can only describe the game by watching the screen (the token stream). The Whisper model itself is the on-field referee, who knows all the rules and has access to internal information.

The suppress_tokens parameter is like a secret directive known only to the referee (e.g., "Player A is not allowed to score today").

• The Referee (Whisper): Sees Player A take a shot. Following the secret directive, he disallows the goal and continues the game according to a backup plan. The official game record (whisper_segments) is coherent and complete.
• The Commentator (whisper-timestamped): Sees Player A take the shot (as it was the highest-probability event) and excitedly yells "GOAL!" (simulating the creation of a segment). But what happens next (the referee's call) makes no sense to him, and his commentary (timestamped_word_segments) quickly becomes disconnected from the actual events of the game.

Recommendations

1. Use suppress_tokens with caution: Try to use it only to suppress regular vocabulary words you don't want (e.g., profanity, filler words like "um" or "ah"). NEVER use it to suppress Whisper's special control tokens, especially the timestamp tokens (IDs >= 50364).
2. Switch to "Naive Mode" as an alternative: If you absolutely must suppress tokens that might cause issues, you can disable the efficient mode by setting naive_approach=True. This mode performs two full inference passes: the first calls the standard model.transcribe to get the "ground truth" text and segments, and the second aligns the audio with that text. While less efficient, it's far more robust because it eliminates the "simulation vs. reality" conflict.

[chenkigba]

In my eval metircs, when I use my own vad segments, there is no difference with suppress_tokens or not.

So it's acceptable. Avoid this by not using suppress_tokens.