[CV2-6066] 4X workerprocesses from 16 to 64 ...

[skyemeedan]

• Adjust the gunicorn configuration to 4X worker processes from 16 to 64 and also adding max-requests and jitter to restart process to reduce impact of memory growth

https://docs.gunicorn.org/en/stable/settings.html

--max-requests INT
Any value greater than zero will limit the number of requests a worker will process before automatically restarting. This is a simple method to help limit the damage of memory leaks.

--max_requests_jitter INT
The maximum jitter to add to the max_requests setting.The jitter causes the restart per worker to be randomized by randint(0, max_requests_jitter). This is intended to stagger worker restarts to avoid all workers restarting at the same time.

• also update CODEOWNERS

Reference: https://meedan.atlassian.net/browse/CV2-6066

[dmou]

ooh big jump. what's the hypothetical impact of this change? is it greater request concurrency? decreased average memory usage?

[skyemeedan]

ooh big jump. what's the hypothetical impact of this change? is it greater request concurrency? decreased average memory usage?

when running with previous settings (16 processes) and a less aggressive test (25 parallel connections vs 100) it seemed like it was finally able to handle the load without the task being marked unhealthy, but still saw some errors. after two days, it is using 80% memory. I'm expecting that this 4X increase will make OOM very likely (perhaps mitigated by the other args) and if we see that, will back it off again.