How do you handle streaming reliability with LiteLLM?

[jamesleeht]

Thanks for the great work on the library!

There are multiple issues that can happen with streaming which LiteLLM helps to handle:

1. Timeout - litellm handles this quite nicely, we can set the stream_timeout parameter.
2. Content filtering - litellm also handles this by looking for the content filter stop reason at the end of the content and throwing a ContentPolicyViolationError. We can also set our own custom string checks to catch the cases outside of the stop reason.

There's an issue I'm not sure how to catch - when the stream cuts off mid sentence does that surface as a specific stop_reason?

Does LiteLLM count it as a stream completion only if it gets a "stop" stop_reason? Couldn't find it in the docs. Would appreciate any help.