Docker Support

[mcharytoniuk]

Continuation of the discussion from #11

[bodybreaker]

I'm working with Docker-compose.
First, I created a llama.cpp docker image file containing the paddler agent and a separate paddler balancer image and configured them to communicate with each other using Docker-compose.

My question is, is it efficient to have multiple llama.cpp containers on only one physical server and loadbalance them with paddler?

Or is it better to have llama.cpp servers on multiple physical servers and load-balance them with paddler?

[mcharytoniuk]

My question is, is it efficient to have multiple llama.cpp containers on only one physical server and loadbalance them with paddler?

Having multiple servers with llama.cpp can make predicting resource usage easier because you can use them all to handle just a single llama.cpp instance, so I would go with that if possible.

Using multiple llama.cpp instances on the same server is also fine, but you need to be really careful with managing the server resources, so they are divided between llama.cpp instances in a way that makes their usage predictable.

You probably also need to tune llama.cpp parameters with llama-bench to predict how your server will perform in both scenarios.

I am using Paddler with Auto Scaling groups and designed it to register and unregister hosts while the balancer runs (depending on the current demand), so it shines the most in that scenario. That applies to, for example, cloud providers and Kubernetes horizontal autoscaling.

So, the short answer is that it depends, but using multiple servers should be easier to manage because of resource isolation.

[Phate334]

Hi, thanks for the great project!

I created a docker compose file to make it easier to run Paddler using Docker:
https://github.com/Phate334/paddler-docker/blob/main/compose.yaml

It works well for my needs, but I was wondering if you have any plans to publish an official Docker image to GitHub Packages (or Docker Hub) in the future. That would simplify the setup even further and help more users get started quickly.

Thanks again for your work!

[Propfend]

Thanks for using Paddler. I am happy you discovered it!

Yes, in the next release we will publish docker images in GitHub registry. If you have any ideas or want to improve the docker image then let us know.

[mcharytoniuk]

@Phate334 If you would like to open up a PR and help us with that, I'd really appreciate it :)