Is it possible to do continuous batching with an openai ChatCompletion compatible interface?

[msugiyama57]

The following code shows an example of doing Continuous batching, can it be done in oepnai ChatCompletion format?
https://github.com/vllm-project/vllm/blob/main/examples/offline_inference.py

[simon-mo]

The OpenAI server automatically batches concurrent requests already, just try it with concurrent requests using any OpenAI compatible clients!