Streaming via the OpenAI API?

[ericmjl]

@brandonwillard and @rlouf, 👋 from me 😄! I've been closely studying outlines, and I've been wondering whether outlines will support streaming from the OpenAI API?

Mostly asking out of curiosity!

[rlouf]

We thought about turning the new Sequence API into a Python generator, in which case it would indeed make sense to include the OpenAI streaming API.

Would you have a use for it?

[ericmjl]

Yes, one of the things I'm considering is to selectively replace LangChain with Outlines in some parts of the LlamaBot codebase. For that, the streaming API improves the ergonomics of interacting with LLMs, especially when large responses need to be composed by the LLM.