Stopping a stream with Langchain LlamCpp in python

[calypset]

Hi, I am looking to stop a stream that is ongoing for any given reason. I set up a stream with the handler as follows, I have a queue and a thread that manages downstream. I have been trying a few things but so far unsucessful. I figured I could pass a stop signal as a token but unsure how.

The callback class is:

from langchain.callbacks.base import BaseCallbackHandler

class QueueCallback(BaseCallbackHandler):
    def __init__(self, q):
        self.q = q

    def cancel(self, q):
        return self.q.empty()

    def on_llm_new_token(self, token: str, **kwargs: Any) -> None:
        self.q.put(token)

    def on_llm_end(self, *args, **kwargs: Any) -> None:
        return self.q.empty()

And model loaded:

from langchain_community.llms import LlamaCpp

llm = LlamaCpp(
        model_path=model_source,
        **params,
        stop=['</s>', '###'], 
        callbacks=[QueueCallback(q)]
)

Anyone has an idea on how to achieve such hard stream stop seamlessly?

PS: Llama.cpp is fantastic!

[reddiamond1234]

I am catching a Keyboard Interrupt exception (ctrl + c)...

[calypset]

I guess my question is how to tell the Llm to stop streaming the answer

[reddiamond1234]

well there isn't

[calypset]

So, then I guess it is true that once a stream from LlamaCpp has started, there is no way to stop it, and the program needs to wait until the stream ends before invoking the chain once again

[reddiamond1234]

As far as I know