How to interrupt streaming output?

[yamikumo-DSD]

Hi, I have a question about interruption of streaming.

How can I safely interrupt streaming?

I want to define stopping criteria to check if an output matches some regex.
Since llama_cpp models don't accept that behavior, I'm using streaming, and at every steps, the concatenated outputs are underwent re.search, and if matches, I break the loop;

    output = ''
    for token in model(prompt, stream=True):
        output += token
        
        match = re.search(pattern, output)
        if match:
            yield output
            break 
        else:
            yield output

However, current llama_cpp implementation doesn't accept the usage.
For example, I'm currently using LlamaCache to save states, and the saving process happens after completion.
When I terminate the generation before it completes, the saving won't happen.
I'm currently copy-n-pasting code to save states, but it depends on llama_cpp's implementation.

So I wanna know how to safely force the model to complete the generation.

Thank you.

[woheller69]

There is a pull request for this, unfortunately not merged:
#733

[yamikumo-DSD]

There is a pull request for this, unfortunately not merged: #733

Ah, it looks old PR, and I cannot get why it's abandoned for long time.