Paddler: open source stateful load balancer custom-tailored for llama.cpp

[mcharytoniuk]

Hello! : )

I finished a new project recently. I needed a load balancer specifically tailored for the llama.cpp that considers its specifics (slots usage, continuous batching). It also works in environments with auto-scaling (you can freely add and remove hosts)

Let me know what you think. Thank you all for creating and maintaining llama.cpp!

PS. I called it "paddler" because I wanted to use Raft protocol initially, but in the end, it was unnecessary. I kept the name, though. :)

Repo:
https://github.com/distantmagic/paddler

[wijjj]

Looks interesting. Thanks for sharing!