File descriptors leak

[d-mankowski-synerise]

ztunnel version: 1.26.3, out of the box config

On one of our node pools, where ingress-controllers reside (so, tons of incoming connections) we have noticed something that looks like ztunnel's file descriptors/sockets leak.

root@aks-ingress6-10369747-vmss00000L:~# cd /proc/$(pgrep ztunnel)
root@aks-ingress6-10369747-vmss00000L:/proc/9569# grep 'Max open files' limits
Max open files            1048576              1048576              files
root@aks-ingress6-10369747-vmss00000L:/proc/9569# cd fd
root@aks-ingress6-10369747-vmss00000L:/proc/9569/fd# ls -la | grep socket -c
383532
root@aks-ingress6-10369747-vmss00000L:/proc/9569/fd# ss -s
Total: 389
TCP:   391116 (estab 31, closed 391051, orphaned 198, timewait 185)

As shown above, there is over 380k sockets, that ztunnel has not actually released (after ~24h since node was spawned). If my understanding is correct, the connection was closed on TCP level, yet ztunnel has never called close() to free up the file descriptor.

ztunnel memory usage:

Socket metrics from node:

[Stevenjin8]

Thanks! Are you using istio for ingress too? Just to help me replicate.

[Stevenjin8]

I tried replicating locally with kind, but everything seems good. Could you run ss -s in the ztunnel network namespace?

nsenter -t 9569 -n
ss -s

[d-mankowski-synerise]

Thanks! Are you using istio for ingress too? Just to help me replicate.

nope, we are not using any istio's L7 features at all

root@aks-ingress6-10369747-vmss00000L:~# pgrep ztunnel
9569
root@aks-ingress6-10369747-vmss00000L:~# nsenter -t 9569 -n
root@aks-ingress6-10369747-vmss00000L:~# ss -s
Total: 9
TCP:   425088 (estab 1, closed 425083, orphaned 238, timewait 8)

Transport Total     IP        IPv6
RAW	  0         0         0
UDP	  0         0         0
TCP	  5         2         3
INET	  5         2         3
FRAG	  0         0         0

I tried replicating locally with kind, but everything seems good.

I wouldn't be surprised if some setting in ingress-controller is causing this abnormal ztunnel behavior, the nginx config has more than 650k lines :D

[Stevenjin8]

@d-mankowski-synerise yeah I hate to push this to someone else, but this does seem like an nginx-ingress issue.

[Stevenjin8]

@d-mankowski-synerise wait, how do you know its ztunnel that's leaking, not nginx-ingress (or ingress-nginx, I never know which is which)

[d-mankowski-synerise]

@d-mankowski-synerise wait, how do you know its ztunnel that's leaking, not nginx-ingress (or ingress-nginx, I never know which is which)

The problem started to occur after we deployed ambient and added ingress controller's namespace to the mesh. After excluding it from the mesh - things got back to normal. That's a first hint. The other would be location of the sockets - sockets in directory /proc/<ztunnel PID>/fd belong to the ztunnel process

root@aks-ingress6-10369747-vmss00000L:~# for pid in $(pgrep nginx); do ls -la /proc/$pid/fd | grep socket -c; done
5
21
14
15
3
5
21
13
12
3
5
37
2015
2021
1925
5
1910
root@aks-ingress6-10369747-vmss00000L:~# for pid in $(pgrep ztunnel); do ls -la /proc/$pid/fd | grep socket -c; done
509345
root@aks-ingress6-10369747-vmss00000L:/proc/$(pgrep ztunnel)/fd# ls -l | head
lrwx------ 1 root 1337 64 Sep  9 14:11 0 -> /dev/null
l-wx------ 1 root 1337 64 Sep  9 14:11 1 -> pipe:[60443]
lrwx------ 1 root 1337 64 Sep 10 22:27 10 -> anon_inode:[eventpoll]
lrwx------ 1 root 1337 64 Sep 10 22:27 100 -> socket:[5467871]
lrwx------ 1 root 1337 64 Sep 10 22:27 1000 -> socket:[1465328]
lrwx------ 1 root 1337 64 Sep 10 22:27 10000 -> socket:[2592726]
lrwx------ 1 root 1337 64 Sep 10 22:27 100000 -> socket:[18334609]
lrwx------ 1 root 1337 64 Sep 10 22:27 100001 -> socket:[16381365]
lrwx------ 1 root 1337 64 Sep 10 22:27 100002 -> socket:[16381367]

and 1337 is the default GID of ztunnel process

❯ kubectl get pod -o yaml ztunnel-zxvlc | grep runAs
      runAsGroup: 1337

[d-mankowski-synerise]

@d-mankowski-synerise yeah I hate to push this to someone else, but this does seem like an nginx-ingress issue.

I mentioned the ingress-controller because it might add some nuances or quirks in how connections are handled, but ultimately, from the kernel's perspective, it is the ztunnel that created the sockets and never closed them (even though the connections were closed)

[Stevenjin8]

Right, ztunnel has a bunch of open FDs... forgot about that

[Stevenjin8]

@d-mankowski-synerise Ok I tried replicating this today on an AKS cluster with ingress nginx installed and no luck... I used the basic configuration to install ingress nginx. Deployed httpbin, labeled both the httpbin and nginx namespace to enable ambient.

I make a pretty bare bones ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-httpbin
spec:
  rules:
  - host: httpbin
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 8000
  ingressClassName: nginx

and slammed it with curl. I saw the number of ztunnel fds go up to about 42k, but it went back done to 29 after a few seconds.

Do you have anything else I could use to replicate? How are you deploying ingress nginx? what kubernetes images are you using? what nodes? Or best would be some way to reproduce it.

[d-mankowski-synerise]

Kubernetes version is 1.31.8, node pool is using Standard_F8s_v2 VM series with AKSUbuntu-2204gen2containerd-202508.11.0 as the node image, all other settings are pretty much default (i.e. as set by azurem terraform provider).

As for the ingress-nginx, it is the custom image based on registry.k8s.io/ingress-nginx/controller:v1.12.3 with a bunch of additional self-compiled modules and a lot of LUA scripts. I am afraid that replicating this issue might be very challenging, as there is plenty of factors (custom image, lua scripts, complex nginx config - none of which, for obvious reasons, I can share) - I will try to play with settings on our staging environment, maybe will get lucky.

There are some things that I have noticed, though:

• this problem does not occur for ingress-nginx serving traffic sent through Cloudflare's WAF (even though ingress-nginx configuration is the same in case of ingress-nginx instances that receive requests with no WAF in path)
• when (the only on a node) ingress-nginx pod was evicted (due to lack of ephemeral space on disk, most likely caused by tons of FDs), ztunnel closed hanging FDs:

ztunnel memory usage:

sockets on node:

If I find anything new, I will let you know

[Stevenjin8]

when (the only on a node) ingress-nginx pod was evicted (due to lack of ephemeral space on disk, most likely caused by tons of FDs), ztunnel closed hanging FDs:

Are you sure that the ztunnel pod isn't getting evicted too? Something isn't adding up.

Also, I'm curious to see what state the sockets in the ztunnel netns. Do you think you could do

cd /proc/<ztunnel pid>/fd
ls -l | grep socket -c
nsenter -t <ztunnel pid> -n ss -t

The output might be huge, but im mostly interested in any patterns (e.g. if there are a bunch of half closed sockets, etc).

Finally, I'm curious to see what the socket state is for nginx's netns. Maybe there is some multiprocessing that causes the fds to appear under a different pid?

for pid in $(pgrep nginx); do nsenter -t $pid -n ss -s | grep socket -c; done
# or
for pid in $(pgrep nginx); do nsenter -st $pid -n ss -s | grep socket -c; done

Again, summarize/redact output as you need. I'm mostly curious about the state

[d-mankowski-synerise]

Are you sure that the ztunnel pod isn't getting evicted too? Something isn't adding up.

yup, I am sure. Ztunnel was up and running all the time:

while the ingress-nginx pod was evicted :

One thing that I have noticed, is that sockets are being modified quite frequently:

root@aks-ingress6-10369747-vmss000026:/proc/9774/fd# pgrep ztunnel
9774
root@aks-ingress6-10369747-vmss000026:/proc/9774/fd# date
Fri Sep 12 14:40:00 UTC 2025
root@aks-ingress6-10369747-vmss000026:/proc/9774/fd# ls -lt | grep 'Sep 12' |  awk '{print $8}' | sort | uniq -c
 199773 13:54
 210371 13:55

so it doesn't even look like ztunnel does not close old sockets, but is using more and more of them over time (ls date shows last modification time), which does not match traffic pattern (which is close to a sine wave). Although, I wouldn't be surprised, if a sheer quantity of short-lived connections (more than a billion request a day, spread over ~35 nodes from millions of unique IP addresses) is triggering some caching mechanism in ztunnel (if there is any) to grow indefinitely.

I did run tcplife for 10s (it shows summary of each tcp connection from establish to close)

root@aks-ingress6-10369747-vmss00001B:~# timeout 10s /usr/share/bcc/tools/tcplife -p $(pgrep ztunnel) > tcplife
root@aks-ingress6-10369747-vmss00001B:~# cat tcplife | grep ztunnel -c
305

sorting by duration of connections (in milliseconds):

0 – 10: 4
10 – 100: 238
100 – 500: 45
500 – 1000: 15
1000 – 2000: 2
2000 – 5000: 1

170 different source IPs.

Another weird thing is that lsof does not print the state of TCP connection

root@aks-ingress6-10369747-vmss00001B:~# lsof -p 8942 -iTCP -P > lsof
root@aks-ingress6-10369747-vmss00001B:~# cat lsof | grep ztunnel | awk '{print $1,$2,$3,$5,$6,$7,$9,$10,$11}' | sort | uniq -c
      1 ztunnel 8942 root CHR 1,3 0t0 /dev/null
      2 ztunnel 8942 root DIR 0,290 4096 /
      2 ztunnel 8942 root FIFO 0,13 0t0 pipe
      1 ztunnel 8942 root REG 0,290 15897600 /usr/local/bin/ztunnel
      1 ztunnel 8942 root REG 0,290 790212 (path dev=8,1, inode=5019)
      1 ztunnel 8942 root REG 0,290 790232 (path dev=8,1, inode=5025)
      1 ztunnel 8942 root REG 0,290 790264 (path dev=8,1, inode=16405)
      1 ztunnel 8942 root REG 0,290 790281 (path dev=8,1, inode=5028)
      3 ztunnel 8942 root REG 0,4 0 net
      2 ztunnel 8942 root a_inode 0,14 0 [eventfd]
      4 ztunnel 8942 root a_inode 0,14 0 [eventpoll]
  46304 ztunnel 8942 root sock 0,8 0t0 protocol: TCP
  46393 ztunnel 8942 root sock 0,8 0t0 protocol: TCPv6
      2 ztunnel 8942 root sock 0,8 0t0 protocol: UDP
      2 ztunnel 8942 root sock 0,8 0t0 protocol: UDPv6
      1 ztunnel 8942 root sock 0,8 0t0 protocol: UNIX
      4 ztunnel 8942 root sock 0,8 0t0 protocol: UNIX-STREAM

which probably indicates, that it is an unbounded/unused socket.

Answering your questions,

The output might be huge, but im mostly interested in any patterns (e.g. if there are a bunch of half closed sockets, etc).

root@aks-ingress6-10369747-vmss00001B:~# cd /proc/$(pgrep ztunnel)/fd
root@aks-ingress6-10369747-vmss00001B:/proc/8942/fd# ls -l | grep socket -c
92590
root@aks-ingress6-10369747-vmss00001B:/proc/8942/fd# nsenter -t 8942 -n ss -t
State                   Recv-Q                   Send-Q                                     Local Address:Port                                         Peer Address:Port                    Process
ESTAB                   0                        0                                           172.26.52.61:35024                                       10.244.61.111:15012
ESTAB                   0                        0                                           172.26.52.61:45652                                       10.244.61.111:15012
root@aks-ingress6-10369747-vmss00001B:/proc/8942/fd#

Again, summarize/redact output as you need. I'm mostly curious about the state

root@aks-ingress6-10369747-vmss00001B:~# for pid in $(pgrep nginx); do nsenter -t $pid -n ss -s; done
Total: 94193
TCP:   101004 (estab 47861, closed 7564, orphaned 23, timewait 6731)

Transport Total     IP        IPv6
RAW	  0         0         0
UDP	  2         1         1
TCP	  93440     46853     46587
INET	  93442     46854     46588
FRAG	  0         0         0

Total: 94193
TCP:   101004 (estab 47859, closed 7564, orphaned 24, timewait 6731)

Transport Total     IP        IPv6
RAW	  0         0         0
UDP	  2         1         1
TCP	  93440     46853     46587
INET	  93442     46854     46588
FRAG	  0         0         0

Total: 94192
TCP:   101003 (estab 47859, closed 7564, orphaned 23, timewait 6731)

Transport Total     IP        IPv6
RAW	  0         0         0
UDP	  2         1         1
TCP	  93439     46853     46586
INET	  93441     46854     46587
FRAG	  0         0         0

Total: 94192
TCP:   101003 (estab 47859, closed 7564, orphaned 23, timewait 6731)

Transport Total     IP        IPv6
RAW	  0         0         0
UDP	  2         1         1
TCP	  93439     46853     46586
INET	  93441     46854     46587
FRAG	  0         0         0

Total: 94192
TCP:   101003 (estab 47859, closed 7564, orphaned 23, timewait 6731)

Transport Total     IP        IPv6
RAW	  0         0         0
UDP	  2         1         1
TCP	  93439     46853     46586
INET	  93441     46854     46587
FRAG	  0         0         0

Total: 94192
TCP:   101003 (estab 47859, closed 7564, orphaned 23, timewait 6731)

Transport Total     IP        IPv6
RAW	  0         0         0
UDP	  2         1         1
TCP	  93439     46853     46586
INET	  93441     46854     46587
FRAG	  0         0         0

Total: 94192
TCP:   101003 (estab 47859, closed 7564, orphaned 23, timewait 6731)

Transport Total     IP        IPv6
RAW	  0         0         0
UDP	  2         1         1
TCP	  93439     46853     46586
INET	  93441     46854     46587
FRAG	  0         0         0

Total: 94192
TCP:   101003 (estab 47859, closed 7564, orphaned 23, timewait 6731)

Transport Total     IP        IPv6
RAW	  0         0         0
UDP	  2         1         1
TCP	  93439     46853     46586
INET	  93441     46854     46587
FRAG	  0         0         0

Total: 94192
TCP:   101003 (estab 47859, closed 7564, orphaned 23, timewait 6731)

Transport Total     IP        IPv6
RAW	  0         0         0
UDP	  2         1         1
TCP	  93439     46853     46586
INET	  93441     46854     46587
FRAG	  0         0         0

[Stevenjin8]

ok, so one thing that I forgot to mention is that ztunnel creates HBONE connection pools from the nginx pod network namespace. That why when we look at /proc//fd there are a bunch of things, but if we ss -s in the ztunnel network ns, there aren't many connections. ztunnel has fds for those connections, but they exist in a different netns.

Anyways, since those sockets exist in the nginx network ns, it makes sense that they disappear after the nginx pod gets evicted.

Some more thoughts.

ztunnel tries to tunnel logical tcp connections over HBONE an hbone connection, but might open many HBONE connections to the same destination if there is a lot of traffic. If an HBONE connection is unused for a while, it gets closed. Could it be the case that the sheer amount of requests you have is preventing this from happening? You can play with connection pooling configurations here by setting environment variables in your ztunnel daemonset (note that for durations, you write the like 5s or 3m).

The other thing is that ingress nginx doesn't reuse upstream connections by default (I believe). Are you configuring ingress nginx to reuse/pool upstream tcp connections? If so, could you share that configuration?

EDIT: could you also look at ztunnel metrics? It shows connections opened and closed

[d-mankowski-synerise]

You can play with connection pooling configurations here by setting environment variables in your ztunnel daemonset (note that for durations, you write the like 5s or 3m).

This month we will be splitting one global ztunnel daemonset to 1 DS per 1 node pool, and also using OnDelete update strategy, so it will be much easier (and without downtime) to change ztunnel parameters. Right now, changing podtemplate would reset all connections in the cluster, hence, impact production. Tuning ztunnel's connection pooling has to wait a little bit, unfortunately

The other thing is that ingress nginx doesn't reuse upstream connections by default (I believe)

It does, in nginx.conf we have:

	upstream upstream_balancer {
		balancer_by_lua_file /etc/nginx/lua/nginx/ngx_conf_balancer.lua;

		keepalive 1024;
		keepalive_time 25s;
		keepalive_timeout  25s;
		keepalive_requests 10000;
	}

EDIT: could you also look at ztunnel metrics? It shows connections opened and closed

from nodes where I see growing number of sockets

I also checked how number of sockets changed after excluding all ingress namespaces from the mesh (we actually did it in our another production cluster)