Need a native queue

[Cyberes]

Why does LiteLLM not have a queue to limit concurrent requests per-model?

[Cyberes]

Made my own: https://github.com/Cyberes/litellm-queue

Still would like native support.

[krrishdholakia]

We do this today -

litellm/litellm/router.py

Line 1074 in 3984962

async with rpm_semaphore:

We run requests inside a semaphore to prevent models from being overwhelmed.

You can set the max parallel request value https://docs.litellm.ai/docs/routing#max-parallel-requests-async, if this is not set we use the rpm limit set as a sanity check

For a global queue, you can also try out scheduler - https://docs.litellm.ai/docs/scheduler