Kill old connections when new connections coming in with RPM policy

[James4Ever0]

Many local LLM services are not responsive when multiple requests coming in. Every request takes a long time to finish and sometimes indefinitely.

I want to resolve this problem by kicking out old connections when new connections coming in.

I have considered other implementations like using Python with proxy.py or Nginx with Lua scripts to proxy these services and manage connections according to the policy mentioned above.

In LiteLLM, if this feature is implemented, it should be configurable as a key for every upstream model entry.

Similar implementations:

vllm-project/vllm#4513

[krrishdholakia]

Hey @James4Ever0 how would you want this to work on litellm?

[James4Ever0]

I would like to do this in system level using ss or iptables, and if not working properly, implement it in LiteLLM.