RAM usage climbing, should I have a cron job to restart nginx periodically?

[adamJLev]

Hi there,

big fan of this project, its been working out very well for me!
quick q- I noticed that over time the server RAM usage goes up slowly.
Is this normal? I'm using the base nginx.conf with some small modifications (e.g. using nginx disk cache instead of memory cache).

Thanks guys!

[root@0b8f96506c8a imagesweserv]# cat /proc/meminfo

MemTotal:       31821952 kB
MemFree:        17520340 kB
MemAvailable:   28377172 kB
Buffers:          179768 kB
Cached:         10709368 kB
SwapCached:            0 kB
Active:          2846256 kB
Inactive:       10695384 kB
Active(anon):    2733396 kB
Inactive(anon):        0 kB
Active(file):     112860 kB
Inactive(file): 10695384 kB
Unevictable:        4000 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:                 4 kB
Writeback:             0 kB
AnonPages:       2528276 kB
Mapped:           237780 kB
Shmem:             80892 kB
KReclaimable:     511360 kB
Slab:             624356 kB
SReclaimable:     511360 kB
SUnreclaim:       112996 kB
KernelStack:        4880 kB
PageTables:        20360 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    15910976 kB
Committed_AS:    1802396 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       19228 kB
VmallocChunk:          0 kB
Percpu:             3584 kB
HardwareCorrupted:     0 kB
AnonHugePages:   2256896 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
Unaccepted:            0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      206248 kB
DirectMap2M:     8439808 kB
DirectMap1G:    25165824 kB

[kleisauke]

Sounds like a memory fragmentation issue. Are you using Docker? If so, is jemalloc enabled there?

images/docker/Dockerfile

Lines 74 to 75 in 4133e2f

	# Use jemalloc on glibc-based Linux systems to reduce the effects of memory fragmentation
	LD_PRELOAD=/usr/lib64/libjemalloc.so

images/docker/README.md

Lines 48 to 49 in 4133e2f

	# Check jemalloc configuration
	docker exec -e MALLOC_CONF="stats_print:true,stats_print_opts:mdablxeh" weserv bash

The pre-built Docker image at ghcr.io/weserv/images:5.x uses jemalloc to prevent memory fragmentation in long-running, multi-threaded, glibc-based Linux processes. See lovell/sharp#955 to learn more about memory allocation and why the choice of a memory allocator is important.

For the same reason Redis ships with jemalloc by default since version 2.4.

Since we introduced the specially encoded data types Redis started suffering from fragmentation. We tried different things to fix the problem, but basically the Linux default allocator in glibc sucks really, really hard. […] Every single case of fragmentation in real world systems was fixed by this change, and also the amount of memory used dropped a bit.

[kleisauke]

To answer the question in the issue title, I think restarting nginx via a cron job is unnecessary.

If you've already tried this and noticed an improvement in memory usage, it might indicate an issue with the proxy cache. Did you also modify the following nginx configuration?:

images/ngx_conf/imagesweserv.conf

Lines 1 to 3 in 4133e2f

	# Please adjust cache size (max_size=250m) to a value that you can accommodate in RAM! Advised values: 2 GB RAM: 500m, 4 GB RAM: 1g, 8 GB RAM: 2g, etc..
	proxy_cache_path /dev/shm/proxy_cache inactive=8h levels=1:2 keys_zone=images:512m max_size=250m loader_files=5000 use_temp_path=off;

[adamJLev]

Thanks for the info, Im looking into jemalloc now. But I assume its already on because my dockerfile is basically just inheriting from the standard Docker file for this project right?

Here's my Dockerfile

FROM ghcr.io/weserv/images:sha-4133e2f

STOPSIGNAL SIGQUIT

COPY imagesweserv.conf /etc/nginx/imagesweserv.conf
COPY public/* /var/www/imagesweserv/public/

EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

and here's the modifications I made to the nginx conf, mainly to increase the timeouts and also to use disk cache instead of memory cache:

weserv_process_timeout 95s;
weserv_max_size 300m;
weserv_max_pages 0;
weserv_limit_input_pixels 0;
weserv_limit_output_pixels 0;
weserv_savers jpg png webp json;

# Disk cache: grows to fill disk but ensures 4GB space free, caches resized images up to 1 year
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=images:64m min_free=4g inactive=1y use_temp_path=off;

proxy_cache_min_uses 1;
proxy_read_timeout 95s;
proxy_send_timeout 95s;
proxy_buffers 256 8k;
proxy_buffering on;

# Enable proxy cache lock to make sure we only process the same url request once at a time.
proxy_cache_lock on;
proxy_cache_lock_timeout 95s;

thanks for your insight, much appreciated!

[kleisauke]

jemalloc should already be enabled if you're inheriting from the pre-built Docker image. The custom nginx config changes looks fine, except for the following lines:

weserv_limit_input_pixels 0;
weserv_limit_output_pixels 0;

You should be careful with that, especially if you're processing untrusted input. There are PNG decompression bombs available that could burn all the available memory.

[adamJLev]

yeah, probably should set a sensible number there. thanks for the tip.

for the "memory growth" issue, jemalloc I guess cant be the issue since its based on the pre-built docker image.
for lack of a better solution, I added a cron job that runs /usr/sbin/nginx -s reload once a day and that seems to fix the issue. thanks for the help!

jemalloc should already be enabled if you're inheriting from the pre-built Docker image. The custom nginx config changes looks fine, except for the following lines:

weserv_limit_input_pixels 0;
weserv_limit_output_pixels 0;

You should be careful with that, especially if you're processing untrusted input. There are PNG decompression bombs available that could burn all the available memory.

[kleisauke]

I'll try to reproduce this with long-running Linux processes (> 1 day), as Certbot reloads the Nginx configuration daily on our servers, which may have masked an underlying memory issue.

Note that you could also fine-tune jemalloc (via the MALLOC_CONF environment variable); see e.g. https://github.com/jemalloc/jemalloc/blob/dev/TUNING.md for details. Unfortunately, jemalloc hasn't had a release in over 2 years.

[adamJLev]

awesome, yeah that sounds like a good test! keep us updated :)

for now I've been periodically restarting the servers via a cron job that calls my cloud hosting API to restart, which works but obv would be ideal to not need that.

thanks 👌🏼

[andrieslouw]

@kleisauke: Should we implement the cron on the pre-built docker? We do it to refresh SSL-certificates regularly (Let's Encrypt is great!), and it is surely not the final answer to memory growth, but maybe it is good practice to reload once in a while?

[kleisauke]

I'm not sure if adding a cron job to the pre-built Docker image is the right approach - it feels a bit unconventional.

We could also look into using mimalloc; I've heard some great things about it compared to jemalloc. A new version of jemalloc is likely to be released in H2, which may cause it to miss the RHEL 10 (and derivatives) release window in May.

[kleisauke]

Commit fb9fd46 updates the Docker base image to Rocky Linux 10. As a result of this update, jemalloc has been temporary disabled since it's currently unavailable in EPEL 10.

In the meantime, the default glibc memory allocator is being used. To help mitigate memory fragmentation, I've set the MALLOC_ARENA_MAX=2 env to reduce the number of memory pools. If you're still experiencing memory fragmentation, I also recommend reducing libvips' concurrency via the VIPS_CONCURRENCY env:

images/ngx_conf/nginx.conf

Lines 12 to 15 in dd2e828

	# The number of worker threads libvips should use
	# for image evaluation. If not set, the number of
	# available CPU cores is used by default.
	env VIPS_CONCURRENCY;

Note that I'm still evaluating the use of mimalloc, as jemalloc's active upstream development has unfortunately ended, see:
https://jasone.github.io/2025/06/12/jemalloc-postmortem/

[adamJLev]

awesome thanks for looking into all this!
I will test out the new image in our dev environment soon and report back with findings

[adamJLev]

Commit fb9fd46 updates the Docker base image to Rocky Linux 10. As a result of this update, jemalloc has been temporary disabled since it's currently unavailable in EPEL 10.

In the meantime, the default glibc memory allocator is being used. To help mitigate memory fragmentation, I've set the MALLOC_ARENA_MAX=2 env to reduce the number of memory pools. If you're still experiencing memory fragmentation, I also recommend reducing libvips' concurrency via the VIPS_CONCURRENCY env:

images/ngx_conf/nginx.conf

Lines 12 to 15 in dd2e828

The number of worker threads libvips should use

for image evaluation. If not set, the number of

available CPU cores is used by default.

env VIPS_CONCURRENCY;
Note that I'm still evaluating the use of mimalloc, as jemalloc's active upstream development has unfortunately ended, see: https://jasone.github.io/2025/06/12/jemalloc-postmortem/

@kleisauke looks like the changes are not quite backwards compatible, or something else is broken.
I updated my dockerfile to pull from the newer weserve docker image sha-dd2e828 and redeployed, and now getting these errors on all routes:

"/var/cache/nginx/7/5e/aac51cf2b673cdaf415a752d18a5d5e7.0000000006" failed (13: Permission denied) while reading upstream, client: ::1, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:80/", host: "localhost"
2025-06-23T15:11:07.064234256Z 2025/06/23 17:11:07 [crit] 7#0: *24 open() "/var/cache/nginx/7/5e/aac51cf2b673cdaf415a752d18a5d5e7.0000000007" failed (13: Permission denied) while reading upstream, client: ::1, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:80/", host: "localhost"
2025-06-23T15:11:07.646787845Z 2025/06/23 17:11:07 [crit] 7#0: *25 open() "/var/cache/nginx/7/5e/aac51cf2b673cdaf415a752d18a5d5e7.0000000008" failed (13: Permission denied) while reading upstream, client: 10.0.0.5, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:80/", host: "[redacted]"
2025-06-23T15:11:12.120206212Z 2025/06/23 17:11:12 [crit] 7#0: *26 open() "/var/cache/nginx/7/5e/aac51cf2b673cdaf415a752d18a5d5e7.0000000009" failed (13: Permission denied) while reading upstream, client: ::1, server: _, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:80/", host: "localhost"

before this, in the dockerfile I was running from this prev docker image version:

FROM ghcr.io/weserv/images:sha-1a0ce63

[kleisauke]

@adamJLev It appears that the PUID/PGID for the nginx user has changed in Rocky Linux 10:

$ docker run --rm -it ghcr.io/weserv/images:sha-1a0ce63 id nginx
uid=993(nginx) gid=991(nginx) groups=991(nginx)
$ docker run --rm -it ghcr.io/weserv/images:sha-dd2e828 id nginx
uid=997(nginx) gid=997(nginx) groups=997(nginx)

If /var/cache/nginx is a mounted volume, you'll likely need to run the following inside the new Docker container to update ownership:

$ chown -R nginx:nginx /var/cache/nginx

[adamJLev]

@kleisauke you're a genius, that was it! thank you.
I have the new version running in our staging environment now, will monitor over the next few days for memory usage and see if there's any changes from previous version