How to scale horizontally with Swarm

[ijpatricio]

Hey all,

1. What's the problem that you're trying to solve?

• I want to horizontally scale with Swarm

2. What results are you getting?

• Inferior results with a "4 server Docker Swarm stack" vs "a single weaker VPS (no docker)"

3. What results are you expecting to see?

• Well, I would expect something below but near 4x the amount of requests. Not in speed, but in amount of concurrency.

I'm using serversideup/php:8.2-fpm-nginx to test the approach I'm taking on launching an app.
So, app-wise, it's a laravel app, but still no features, I just have a couple of endpoints and a JS build, in a multi-stage Dockerfile.

The endpoint I use on ApacheBench is a simple

// routes/web.php
Route::get('/check', function () {
    return response()->json('OK', 200);
});

and the AB command:

ab -n 1000 -c 20  http://<SERVER_IP>/check

Relevant ENV variables:

PHP_PM_MAX_SPARE_SERVERS=10
PHP_PM_MAX_CHILDREN=20
PHP_PM_START_SERVERS=5
PHP_PM_MIN_SPARE_SERVERS=1
PHP_PM_CONTROL=dynamic

Using Hetzner, which describe vps as:
CAX11 = arm64, 2 vcpu, 4gb ram
CAX31 = arm64, 8 vcpu, 16gb ram

Results 1:

With Stack1 = 1x CAX11 manager node + 3x CAX31,
3 replicas of webapp

Document Path:          /check
Document Length:        4 bytes

Concurrency Level:      20
Time taken for tests:   36.605 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      1296000 bytes
HTML transferred:       4000 bytes
Requests per second:    27.32 [#/sec] (mean)
Time per request:       732.101 [ms] (mean)
Time per request:       36.605 [ms] (mean, across all concurrent requests)
Transfer rate:          34.58 [Kbytes/sec] received

Results 2:

With Stack1 = 1x CAX11 manager node + 3x CAX31,
12 replicas of webapp

Document Path:          /check
Document Length:        4 bytes

Concurrency Level:      20
Time taken for tests:   25.313 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      1296000 bytes
HTML transferred:       4000 bytes
Requests per second:    39.50 [#/sec] (mean)
Time per request:       506.270 [ms] (mean)
Time per request:       25.313 [ms] (mean, across all concurrent requests)
Transfer rate:          50.00 [Kbytes/sec] received

I did plenty of tests this past days, and I know results will be surprising, but I want a good baseline.
Installing now a Forge VPS, on a CAX31, and check compare.
Yep, as I thought...

Results 3

Single VPS, CAX31, no docker, Forge nginx+fpm default configs

Document Path:          /check
Document Length:        4 bytes

Concurrency Level:      20
Time taken for tests:   8.670 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      1214000 bytes
HTML transferred:       4000 bytes
Requests per second:    115.34 [#/sec] (mean)
Time per request:       173.393 [ms] (mean)
Time per request:       8.670 [ms] (mean, across all concurrent requests)
Transfer rate:          136.75 [Kbytes/sec] received

I have deploy scripts and ansible in place, so any tests you would like to see, just let me know, and I will setup the thing and run it.

I'm right about these results, they don't make sense do they??

[ijpatricio]

I went read again https://serversideup.net/open-source/docker-php/docs/reference/environment-variable-specification

Should I have

PHP_PM_CONTROL=ondemand
PHP_PM_MAX_CHILDREN=20

After changing to this, I get from running ps -aux inside on of the containers:

root           1  0.0  0.0    200    68 ?        Ss   15:12   0:00 /package/admin/s6/command/s6-svscan -d4 -- /run/service
root          18  0.0  0.0    204    68 ?        S    15:12   0:00 s6-supervise s6-linux-init-shutdownd
root          19  0.0  0.0    192     4 ?        Ss   15:12   0:00 /package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B
root          27  0.0  0.0    204    68 ?        S    15:12   0:00 s6-supervise s6rc-fdholder
root          28  0.0  0.0    204    68 ?        S    15:12   0:00 s6-supervise scheduler-worker
root          29  0.0  0.0    204    68 ?        S    15:12   0:00 s6-supervise nginx
root          30  0.0  0.0    204    68 ?        S    15:12   0:00 s6-supervise queue-worker
root          31  0.0  0.0    204    72 ?        S    15:12   0:00 s6-supervise php-fpm
root          32  0.0  0.0    204    68 ?        S    15:12   0:00 s6-supervise s6rc-oneshot-runner
root          45  0.0  0.0   6912  2908 ?        Ss   15:12   0:00 su - webuser -c php /var/www/html/artisan schedule:work
root          46  0.0  0.0    180     4 ?        Ss   15:12   0:00 /package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -
root          47  0.0  0.0   6912  2908 ?        Ss   15:12   0:00 su - webuser -c php /var/www/html/artisan queue:work --timeout=600 --memory=512
webuser       53  0.3  1.4 139132 56096 ?        Ss   15:12   0:00 php /var/www/html/artisan schedule:work
webuser       59  0.2  1.4 138996 55556 ?        Ss   15:12   0:00 php /var/www/html/artisan queue:work --timeout=600 --memory=512
root          99  0.0  0.7 244292 30620 ?        Ss   15:12   0:00 php-fpm: master process (/etc/php/8.2/fpm/php-fpm.conf)
root         130  0.0  0.1   9408  5756 ?        Ss   15:12   0:00 nginx: master process nginx
webuser      134  0.0  0.0   9748  3480 ?        S    15:12   0:00 nginx: worker process
webuser      135  0.0  0.0   9748  2424 ?        S    15:12   0:00 nginx: worker process
root         179  0.0  0.0   6412  1608 pts/0    Rs+  15:14   0:00 ps -aux

So it seems there's even less than before??

[jaydrogers]

Thanks for opening this on GitHub Discussions! As for any set up like this, it gets complicated 🤪

In order to fully diagnose the issue, unfortunately you might have to draw more things out and post your configurations to make this repeatable.

Instead of going through the effort of that -- here's a few things you might want to research.

The VPS servers may be different

Although you buy the same package from the same host, your performance may vary server to server. This may be due to other tenants on the same box as you. I've even seen totally different processors deployed within the same package.

Ensure your servers are in the same datacenter

You want the servers to be as close as possible. Even two servers sitting next to each other will have increased latency compared to a single box.

Consider using static for Process Control

If performance is your #1 priority, you may want to move to static for PHP FPM. This will eliminate any delay of processes starting on a request: https://haydenjames.io/php-fpm-tuning-using-pm-static-max-performance/

You may want to use fpm variation instead of fpm-nginx

The fpm-nginx variation is great for small set ups. It includes a web server and PHP-FPM.

When you scale out, you don't need NGINX on every container instance. Instead, you would want to have a load balancer in front (possibly even run NGINX or Traefik as its own service). Then you would build your PHP apps with the fpm variation -- giving you a lot less weight within the image which will also reduce latency.

Hope this helps! 👍

[ijpatricio]

They are in the same datacenter. I made everything as similar as possible.
I understand what you say, and I agree.

But I think we both agree that 1xSERVER cannot be more performant than 4xSERVER Swarm, right? even if shared etc etc...

I'm not worried at this point to save resources and bring costs down. Just want to have control over the process, then I can fine tune.

ALso, just noticed this, you say static, but docs say otherwise, ondemand

[ijpatricio]

Hey now we're getting somewhere!!

PHP_PM_CONTROL=static
PHP_PM_MAX_CHILDREN=20

I put static like you said, instead of ondemand

[ijpatricio]

So now I figure we're OK with the containers, and Swarm networking is the new bottleneck, that will be solved by Traefik, right??

At least that's what I think.. I'll let you know, also let me know if I'm thinking something wrong

The docs need update, right?? Would you accept PR?

[jaydrogers]

So now I figure we're OK with the containers, and Swarm networking is the new bottleneck, that will be solved by Traefik, right??

Not sure if it would be the Swarm networking or not. Traefik would be the first reverse proxy that would load balance across your containers.

The docs need update, right?? Would you accept PR?
I don't think the docs need an update because I am listing what the defaults are. In the screenshot you highlighted below, ondemand is the default.

I do this to minimize CPU and memory usage (which is great for 99% of apps)

[ijpatricio]

Hey @jaydrogers

It turns out, it's a multitude of stuff. SwarmMode is supposed to do round-robin to a service (layer 3 - IP address). With Traefik we can do that at host name (layer 4) - all good here.
Thing is, after I monitor CPU, and instant network throughput (bmon), I then realized the networking between swarm nodes wasn't fully functioning, not even inside the private network.

In a gist, it depends on the cloud provider we're using.

Regarding this image, and what could be changed, I suggested a change to the docs, because I overlooked or needed to check the official docs.

It would be nice to know there's the static option also, without having to go into the official docs. That was it.

You're doing an awesome job, can't wait to see more and more stuff out in the wild!!

Don't hesitate to hit me up about anything!