Minimal RAM requirements

[tipa]

I am running on a shared hosting environment with limited RAM per user (1.5 GB). I understand that Weblate states 3GB as minimal requirement but I hope to be able to run Weblate nonetheless. The requirements seem to be quite high for just a small website.
I have set CELERY_WORKER_CONCURRENCY = 1 in order to limit the tasks per worker to two (one main celery task and one worker task, as far as I understand). Only then the celery tasks are not killed by the system for exceeding the RAM limit.
Additional settings I have set:

CELERY_WORKER_PREFETCH_MULTIPLIER = 1
CELERY_WORKER_MAX_MEMORY_PER_CHILD = 100000 # normally set to 200000 # in kb
CELERY_WORKER_MAX_TASKS_PER_CHILD = 1 # default: no limit

But those tasks still seem to consume up to 142MB per task, as long as I am interpreting these numbers correctly. Is it normal that these tasks require so much RAM even though Weblate is completely idle / freshly set up?

Even when Weblate is running after setting CELERY_WORKER_CONCURRENCY = 1, as soon as I try to create a component based on a Git Repository, the app gets killed:

[2023-10-23 17:59:32 +0200] [17606] [INFO] Using worker: sync
[2023-10-23 17:59:32 +0200] [17609] [INFO] Booting worker with pid: 17609
[2023-10-23 18:19:34 +0200] [17606] [CRITICAL] WORKER TIMEOUT (pid:17609)
[2023-10-23 18:19:35 +0200] [17606] [ERROR] Worker (pid:17609) was sent SIGKILL! Perhaps out of memory?

This is my celery configuration for supervisord:

[group:celery]
programs=celery-backup,celery-celery,celery-memory,celery-notify,celery-translate,celery-beat
 
[program:celery-backup]
environment=CELERY_WORKER_RUNNING=1,CELERY_APP=weblate.utils,LD_LIBRARY_PATH=%(ENV_HOME)s/usr/local/lib,PGHOST=localhost
command=/home/tipako/weblate/bin/celery worker --hostname 'backup@%%h' --queues=backup
autorestart=false
startsecs=10
stdout_events_enabled=false
stderr_events_enabled=false

[program:celery-celery]
environment=CELERY_WORKER_RUNNING=1,CELERY_APP=weblate.utils,LD_LIBRARY_PATH=%(ENV_HOME)s/usr/local/lib,PGHOST=localhost
command=/home/tipako/weblate/bin/celery worker --hostname 'celery@%%h' --queues=celery
autorestart=false
startsecs=10
stdout_events_enabled=false
stderr_events_enabled=false

[program:celery-memory]
environment=CELERY_WORKER_RUNNING=1,CELERY_APP=weblate.utils,LD_LIBRARY_PATH=%(ENV_HOME)s/usr/local/lib,PGHOST=localhost
command=/home/tipako/weblate/bin/celery worker --hostname 'memory@%%h' --queues=memory
autorestart=false
startsecs=10
stdout_events_enabled=false
stderr_events_enabled=false

[program:celery-notify]
environment=CELERY_WORKER_RUNNING=1,CELERY_APP=weblate.utils,LD_LIBRARY_PATH=%(ENV_HOME)s/usr/local/lib,PGHOST=localhost
command=/home/tipako/weblate/bin/celery worker --hostname 'notify@%%h' --queues=notify
autorestart=false
startsecs=10
stdout_events_enabled=false
stderr_events_enabled=false

[program:celery-translate]
environment=CELERY_WORKER_RUNNING=1,CELERY_APP=weblate.utils,LD_LIBRARY_PATH=%(ENV_HOME)s/usr/local/lib,PGHOST=localhost
command=/home/tipako/weblate/bin/celery worker --hostname 'translate@%%h' --queues=translate
autorestart=false
startsecs=10
stdout_events_enabled=false
stderr_events_enabled=false

[program:celery-beat]
environment=CELERY_WORKER_RUNNING=1,CELERY_APP=weblate.utils,LD_LIBRARY_PATH=%(ENV_HOME)s/usr/local/lib,PGHOST=localhost
command=/home/tipako/weblate/bin/celery beat
autorestart=false
startsecs=10
stdout_events_enabled=false
stderr_events_enabled=false

[nijel]

You can reduce memory usage by having a single Celery worker to process all the queues:

celery worker --beat --queues=celery,notify,memory,translate,backup 

CELERY_WORKER_MAX_MEMORY_PER_CHILD = 100000 will probably lead to a horrible performance, as it will cause the worker to restart after every task.

[tipa]

Thanks for your answer. After some more testing I was able to reduce the amount of tasks (and thus memory) by using the --pool=solo parameter for the celery command.
I assume, the CELERY_WORKER_MAX_MEMORY_PER_CHILD parameter will be unused in that case, as no children are spawned. I have commented out the line in the settings.py file now. Weblate seems to be running quite well now, with much less use of memory. Maybe these findings help other users who also hit memory limits.
Other apps I am running on my server (Nextcloud, Flask) still consume considerably less memory, but I guess that's due to the different technologies and project architecture.

[nijel]

Using --pool=solo makes sense as well. I've added both to our documentation in bb6cc8d.