Triton server prediction streaming interface for LLM

[sepo-eng]

Has anyone in the community attempted to use Triton for streaming tokens, similar to Open AI's Chat GPT?

I've come across the ModelStreamInfer method exposed in the grpc_service.proto interface, but this seems to be responding with all the tokens at once.

[outtanames]

If you just want an inference server solution for this you can try https://github.com/autonomi-ai/nos/tree/main/examples/tutorials/03-llm-streaming-chat. Llama 7B works out of the box locally with a grpc streaming interface and is pretty easy to set up.