ztunnel 1.27.0 steady memory growth (Ambient / Helm) possible leak

[strobil]

Setup

• ztunnel 1.27.0, installed via Helm, profile ambient
• Requests = limits: CPU 4, Mem 4Gi
• Env: ZTUNNEL_WORKER_THREADS=4, IPV6_ENABLED=false, logs warn/JSON

Helm values (trimmed)

profile: ambient
resources:
  requests: { cpu: 4, memory: 4Gi }
  limits:   { cpu: 4, memory: 4Gi }
env:
  ZTUNNEL_WORKER_THREADS: "4"
  IPV6_ENABLED: "false"
logLevel: warn
logAsJson: true

Symptom

• Over ~Sep 05–14, memory for ztunnel pods grows steadily and never returns to baseline

Evidence

• Screenshot attached (“Distribution: Pod usage (bytes, stacked)”).

Expected

• Memory should remain roughly stable for a steady workload.

[keithmattix]

/cc @Stevenjin8 @krinkinmu

[krinkinmu]

@strobil can you share a bit more about this graph, what metric did you use to build it? Is it container_memory_working_set_bytes or something else?

[strobil]

@strobil can you share a bit more about this graph, what metric did you use to build it? Is it container_memory_working_set_bytes or something else?

@krinkinmu
Yes, it's container_memory_working_set_bytes

[strobil]

@krinkinmu please take a closer look at this memory and cpu snapshots - most memory allocations occur during peak hours.

Memory: https://snapshots.raintank.io/dashboard/snapshot/YaYe56CQxBobt0HzH9U9FpI2kpkesveF
CPU: https://snapshots.raintank.io/dashboard/snapshot/nYBlZTCWBii1fg1HhcVlwpQ4nr1DEpAQ

Offtopic: Prior to upgrading from 1.25 to 1.27 we ran into a different issue related to the default number of worker threads. At peak load we observed increased latency because several pods were fully utilizing two cores (see CPU graph for September 2–4). This was not immediately obvious because the Deployment had no CPU limits configured. We later confirmed that ztunnel defaults to two worker threads (reference: https://github.com/istio/ztunnel/blob/master/src/config.rs#L81C1-L81C39).

[Stevenjin8]

#1629

We're tracking a similar issue here (maybe). Do you have any stats on number of file descriptors open?

[krinkinmu]

I tried to reproduce it with jemalloc to grab an allocation profile, but couldn't see memory growing yet. It might be because the behavior onlytriggers with much higher load or because it's a glibc ptmalloc issue (and therefore it does not trigger when I switched to jemalloc).

Leaking file descriptors issue may be relevant here, but I'm going to look at other options, given that @Stevenjin8 has been looking at file descriptors.

[krinkinmu]

I was able to see a one time increase when I put load on ztunnel, what I could not replicate is a further growth after that. It's expected that the amount of memory ztunnel uses does depend on the load it handles (e.g., the number of connections plus some buffer per connection are required), however in all my experiments it grows to a point and then the growth stops.

…

On Mon, 15 Sept 2025 at 15:55, Steven Jin ***@***.***> wrote: I think I was able to replicate: here is the mem profile when ztunnel first starts VmPeak: 490900 kB VmSize: 490900 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 13952 kB VmRSS: 13952 kB VmData: 15988 kB VmStk: 140 kB VmExe: 10432 kB VmLib: 2312 kB VmPTE: 128 kB VmSwap: 0 kB Then, I run fortio load -qps 0 -nocatchup -uniform -payload-size 10000 -t 20s -s 20 -c 100 172.19.255.200/post and get VmPeak: 556148 kB VmSize: 491368 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 29528 kB VmRSS: 19052 kB VmData: 64980 kB VmStk: 144 kB VmExe: 10432 kB VmLib: 2312 kB VmPTE: 216 kB VmSwap: 0 kB — Reply to this email directly, view it on GitHub <#1630 (comment)> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI7Q46TTNWHCEEJJCQF3G33S3HPNBFHORZGSZ3HMVZKMY3SMVQXIZNMON2WE2TFMN2F65DZOBS2YSLTON2WKQ3PNVWWK3TUUZ2G64DJMNZZFAVEOR4XAZNKOJSXA33TNF2G64TZUV3GC3DVMWUTKNJUGM4DCNJSG2BKI5DZOBS2K2LTON2WLJLWMFWHKZNKGM2DCNJTGA4TSNJVVJQXI5DSNFRHK5DFOOKIFJLWMFWHKZNEORZHKZNENZQW2ZN3ORUHEZLBMRPXAYLSORUWG2LQMFXHIX3BMN2GS5TJOR4YFJLWMFWHKZNEORZHKZNENZQW2ZNOO5QXIY3IL5QWG5DJOZUXI6MCUV3GC3DVMWSWS43TOVS2I3TBNVS2W5DIOJSWCZC7OR4XAZMCUV3GC3DVMWUDGNBRGIYDKMRQURXGC3LFVBQWG5DPOJPWSZA> . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

[howardjohn]

ztunnel has growing buffers based on the lifetime bytes sent in a connection. this means with long enough connection you'll always hit it even in low bandwidth connections. This could play a factor here.

We probably should make it only grow if it's actually shown high bandwidth (TBD how exactly)

[krinkinmu]

ztunnel has growing buffers based on the lifetime bytes sent in a connection. this means with long enough connection you'll always hit it even in low bandwidth connections. This could play a factor here.

We probably should make it only grow if it's actually shown high bandwidth (TBD how exactly)

Doesn't that logic already limit the max buffer size to ~256k, e.g., it does not grow unbounded?

I can imagine that the apparent leak is just more and more long-lived connections eventually crossing the buffer growth threshold... Let's say we round up the peak memory usage to 4GiBs and the buffer size per connection is 256KiBs at that point, that would be about 16k active connections.

If we say that bazeline usage is 1GiB (e.g., when there is no load), then that number goes down to something like 12k connections, but still quite a lot.

@strobil by any chance can you try istioctl zc cons | wc -l on a ztunnel with large memory consumption (e.g., istioctl zc cons ztunnel-2vrhw | wc -l) to see how many connections it serves (that would not be long running connections, but all the connections, still might be useful to observe)?

[krinkinmu]

We probably should make it only grow if it's actually shown high bandwidth (TBD how exactly)

I think we should confirm that it's possible that it's a significant contributing factor here before changing anything, but one idea is to not grow if we don't fill the already available buffer in one read in the first place?

[howardjohn]

We probably should make it only grow if it's actually shown high bandwidth (TBD how exactly)

I think we should confirm that it's possible that it's a significant contributing factor here before changing anything, but one idea is to not grow if we don't fill the already available buffer in one read in the first place?

Yeah hyper has a mechanism similar to this. They also shrink buffers, not sure we really want that. But it would be nice to avoid a single big read permanently spiking the size.

[Stevenjin8]

@strobil I also noticed that there are a few ztunnels that use a bunch of memory. Is there anything special about those? e.g. are they on the same node as say a gateway?

[strobil]

@strobil I also noticed that there are a few ztunnels that use a bunch of memory. Is there anything special about those? e.g. are they on the same node as say a gateway?

Those are deployed on master nodes, so they show almost no load.

By the way, do you know if deploying ztunnel pods on master nodes is mandatory? I couldn’t find a clear statement about this in the documentation.

[strobil]

#1629

We're tracking a similar issue here (maybe). Do you have any stats on number of file descriptors open?

I replicated these commands on two worker nodes in our dev cluster.

Details

worker01 ~ # cd /proc/$(pgrep ztunnel)
worker01 /proc/135816 # grep 'Max open files' limits
Max open files            1048576              1048576              files     
worker01 /proc/135816 # cd fd
worker01 /proc/135816/fd # ls -la | grep socket -c
ls: cannot access '26267': No such file or directory
26210
worker01 /proc/135816/fd # ss -s
Total: 1620
**TCP:   41283 (estab 97, closed 41002, orphaned 9, timewait 1299)**

Transport Total     IP        IPv6
RAW	  5         0         5        
UDP	  7         5         2        
TCP	  281       271       10       
INET	  293       276       17       
FRAG	  0         0         0        

worker01 /proc/135816/fd # 

worker02 ~ # cd /proc/$(pgrep ztunnel)
worker02 /proc/3887537 # grep 'Max open files' limits
Max open files            1048576              1048576              files     
worker02 /proc/3887537 # cd fd
worker02 /proc/3887537/fd # ls -la | grep socket -c
11185
worker02 /proc/3887537/fd # ss -s
Total: 744
**TCP:   25507 (estab 41, closed 25378, orphaned 0, timewait 393)**

Transport Total     IP        IPv6
RAW	  3         0         3        
UDP	  1         1         0        
TCP	  129       114       15       
INET	  133       115       18       
FRAG	  0         0         0        

worker02 /proc/3887537/fd # 

Worker 1 has ~1.6× more closed TCP connections than Worker 2, but this doesn’t align with the actual pod memory usage.

ztunnel-xrpn8 running on worker1 has an actual memory consumption of 2.33 GiB
ztunnel-mfnf8 running on worker2 has an actual memory consumption of 460 MiB

The difference in memory consumption is about 5×.
This suggests that closed TCP connections may not be the only issue.

Memory usage on dev cluster - https://snapshots.raintank.io/dashboard/snapshot/etnlLy8MLgd2FyS72prCXKYr2LDy9bV9

[krinkinmu]

And what about number of ztunnel connections on the node with high memory usage?

…

On Mon 15 Sept 2025, 21:07 Mykhailo Alipa, ***@***.***> wrote: #1629 <#1629> We're tracking a similar issue here (maybe). Do you have any stats on number of file descriptors open? I replicated these commands on two worker nodes in our dev cluster. worker01 ~ # cd /proc/$(pgrep ztunnel) worker01 /proc/135816 # grep 'Max open files' limits Max open files 1048576 1048576 files worker01 /proc/135816 # cd fd worker01 /proc/135816/fd # ls -la | grep socket -c ls: cannot access '26267': No such file or directory 26210 worker01 /proc/135816/fd # ss -s Total: 1620**TCP: 41283 (estab 97, closed 41002, orphaned 9, timewait 1299)** Transport Total IP IPv6 RAW 5 0 5 UDP 7 5 2 TCP 281 271 10 INET 293 276 17 FRAG 0 0 0 worker01 /proc/135816/fd # worker02 ~ # cd /proc/$(pgrep ztunnel) worker02 /proc/3887537 # grep 'Max open files' limits Max open files 1048576 1048576 files worker02 /proc/3887537 # cd fd worker02 /proc/3887537/fd # ls -la | grep socket -c 11185 worker02 /proc/3887537/fd # ss -s Total: 744**TCP: 25507 (estab 41, closed 25378, orphaned 0, timewait 393)** Transport Total IP IPv6 RAW 3 0 3 UDP 1 1 0 TCP 129 114 15 INET 133 115 18 FRAG 0 0 0 worker02 /proc/3887537/fd # Worker 1 has ~1.6× more closed TCP connections than Worker 2, but this doesn’t align with the actual pod memory usage. ztunnel-xrpn8 running on worker1 has an actual memory consumption of 2.33 GiB ztunnel-mfnf8 running on worker2 has an actual memory consumption of 460 MiB The difference in memory consumption is about 5×. This suggests that closed TCP connections may not be the only issue. Memory usage on dev cluster - https://snapshots.raintank.io/dashboard/snapshot/etnlLy8MLgd2FyS72prCXKYr2LDy9bV9 — Reply to this email directly, view it on GitHub <#1630 (comment)> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI7Q42Z3JAZRHSDBGDXPCT3S4L7DBFHORZGSZ3HMVZKMY3SMVQXIZNMON2WE2TFMN2F65DZOBS2YSLTON2WKQ3PNVWWK3TUUZ2G64DJMNZZFAVEOR4XAZNKOJSXA33TNF2G64TZUV3GC3DVMWUTKNJUGM4DCNJSG2BKI5DZOBS2K2LTON2WLJLWMFWHKZNKGM2DCNJTGA4TSNJVVJQXI5DSNFRHK5DFOOKIFJLWMFWHKZNEORZHKZNENZQW2ZN3ORUHEZLBMRPXAYLSORUWG2LQMFXHIX3BMN2GS5TJOR4YFJLWMFWHKZNEORZHKZNENZQW2ZNOO5QXIY3IL5QWG5DJOZUXI6MCUV3GC3DVMWSWS43TOVS2I3TBNVS2W5DIOJSWCZC7OR4XAZMCUV3GC3DVMWTTMNBUGI2TOMVENZQW2ZNIMFRXI33SL5UWI> . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .