Upload is broken (9: Bad file descriptor)

[sepich]

What happened:

POST with 10mb body leads to dropped request with error:

2025/02/20 15:24:11 [crit] 517#517: *74177 pread() "/tmp/nginx/client-body/0000000041" failed (9: Bad file descriptor) while sending request to upstream

What you expected to happen:

Http request succeeds.

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):
v1.12.0

Kubernetes version (use kubectl version):
v1.30.8

Environment:

• Cloud provider or hardware configuration: EKS/GKE/on-prem

How to reproduce this issue:

Could be reproduced in kind:

kind create cluster
kubectl apply -f https://kind.sigs.k8s.io/examples/ingress/deploy-ingress-nginx.yaml
kubectl -n ingress-nginx port-forward ingress-nginx-controller-696d4c4c5-x7kl6 8080:80

Create test Ingress accepting uploads:

yaml spec

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
spec:
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - image: inanimate/echo-server
          name: echo
      enableServiceLinks: false
---
apiVersion: v1
kind: Service
metadata:
  name: echo
spec:
  selector:
    app: echo
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
  #    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
  name: echo
spec:
  ingressClassName: nginx
  rules:
    - http:
        paths:
          - backend:
              service:
                name: echo
                port:
                  number: 8080
            path: /
            pathType: ImplementationSpecific

Upload file, see the connection dropped:

dd if=/dev/urandom of=10M bs=1M count=10
curl -d @10M http://localhost:8080 > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to localhost port 8080 after 0 ms: Couldn't connect to server

In logs:

$ k -n ingress-nginx logs --tail=5 ingress-nginx-controller-696d4c4c5-x7kl6

I0220 16:14:51.539815      11 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kube-system", Name:"echo", UID:"eb48935e-1c17-40d5-9563-599b29c8d844", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1177", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
127.0.0.1 - - [20/Feb/2025:16:57:25 +0000] "GET / HTTP/1.1" 200 1779 "-" "curl/8.7.1" 77 0.029 [kube-system-echo-8080] [] 10.244.0.8:8080 1779 0.029 200 3d0c1d9f60cec40e163e0f2713eb9821
2025/02/20 16:57:38 [warn] 573#573: *32687 a client request body is buffered to a temporary file /tmp/nginx/client-body/0000000001, client: 127.0.0.1, server: _, request: "POST / HTTP/1.1", host: "localhost:8080"
2025/02/20 16:57:38 [crit] 573#573: *32687 pread() "/tmp/nginx/client-body/0000000001" failed (9: Bad file descriptor) while sending request to upstream, client: 127.0.0.1, server: _, request: "POST / HTTP/1.1", upstream: "http://10.244.0.8:8080/", host: "localhost:8080"
127.0.0.1 - - [20/Feb/2025:16:57:39 +0000] "POST / HTTP/1.1" 200 3959 "-" "curl/8.7.1" 10363454 1.035 [kube-system-echo-8080] [] 10.244.0.8:8080 3939 0.003 502 73dfb8ab83944211aadfc70382080cd8

Per debug logs it happens after upstream already answered 200 OK, and response is forwarded back to client:

2025/02/20 17:21:31 [debug] 1114#1114: *101569 http body new buf t:1 f:0 0000FFFF802C1690, pos 0000FFFF802C1690, size: 8192 file: 0, size: 0
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http write client request body, bufs 0000FFFF80287290
2025/02/20 17:21:31 [debug] 1114#1114: *101569 add cleanup: 0000FFFF802872A0
2025/02/20 17:21:31 [debug] 1114#1114: *101569 hashed path: /tmp/nginx/client-body/0000000002
2025/02/20 17:21:31 [debug] 1114#1114: *101569 temp fd:12
2025/02/20 17:21:31 [warn] 1114#1114: *101569 a client request body is buffered to a temporary file /tmp/nginx/client-body/0000000002, client: 127.0.0.1, server: _, request: "POST / HTTP/1.1", host: "localhost:8080"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 write: 12, 0000FFFF802C1690, 8192, 0
...
2025/02/20 17:21:31 [debug] 1114#1114: *101569 read: 12, 0000FFFF802C1690, 8192, 360448
...
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http proxy status 200 "200 OK"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http proxy header: "Content-Type: text/plain"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http proxy header: "X-Real-Server: echo-server"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http proxy header: "Date: Thu, 20 Feb 2025 17:21:31 GMT"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http proxy header: "Connection: close"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http proxy header: "Transfer-Encoding: chunked"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http proxy header done
2025/02/20 17:21:31 [debug] 1114#1114: *101569 lua header filter for user lua code, uri "/"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 code cache lookup (key='nhlf_9f4fbd7160880a0dbc1df5df3dfe0b59', ref=-1)
2025/02/20 17:21:31 [debug] 1114#1114: *101569 code cache miss (key='nhlf_9f4fbd7160880a0dbc1df5df3dfe0b59', ref=-1)
2025/02/20 17:21:31 [debug] 1114#1114: *101569 lua capture header filter, uri "/"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 headers more header filter, uri "/"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 HTTP/1.1 200 OK
Date: Thu, 20 Feb 2025 17:21:31 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive
X-Real-Server: echo-server

2025/02/20 17:21:31 [debug] 1114#1114: *101569 write new buf t:1 f:0 0000FFFF80287F70, pos 0000FFFF80287F70, size: 162 file: 0, size: 0
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http write filter: l:0 f:0 s:162
2025/02/20 17:21:31 [debug] 1114#1114: *101569 file cleanup: fd:12
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http proxy filter init s:200 h:0 c:1 l:-1
2025/02/20 17:21:31 [debug] 1114#1114: *101569 tcp_nodelay
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunked byte: 38 s:0
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunked byte: 30 s:1
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunked byte: 30 s:1
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunked byte: 0D s:1
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunked byte: 0A s:3
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunked byte: 57 s:4
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http proxy out buf 0000FFFF80199592 2048
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunked byte: 0D s:5
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunked byte: 0A s:6
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunked byte: 64 s:0
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunked byte: 63 s:1
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunked byte: 62 s:1
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunked byte: 0D s:1
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunked byte: 0A s:3
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunked byte: 2C s:4
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http proxy out buf 0000FFFF80199D99 1879
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http upstream process non buffered downstream
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http output filter "/?"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http copy filter: "/?"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 lua capture body filter, uri "/"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http postpone filter "/?" 0000FFFF80288150
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunk: 2048
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http chunk: 1879
2025/02/20 17:21:31 [debug] 1114#1114: *101569 write old buf t:1 f:0 0000FFFF80287F70, pos 0000FFFF80287F70, size: 162 file: 0, size: 0
2025/02/20 17:21:31 [debug] 1114#1114: *101569 write new buf t:1 f:0 0000FFFF802881F0, pos 0000FFFF802881F0, size: 5 file: 0, size: 0
2025/02/20 17:21:31 [debug] 1114#1114: *101569 write new buf t:0 f:0 0000000000000000, pos 0000FFFF80199592, size: 2048 file: 0, size: 0
2025/02/20 17:21:31 [debug] 1114#1114: *101569 write new buf t:0 f:0 0000000000000000, pos 0000FFFF80199D99, size: 1879 file: 0, size: 0
2025/02/20 17:21:31 [debug] 1114#1114: *101569 write new buf t:0 f:0 0000000000000000, pos 0000AAAABA2B2288, size: 2 file: 0, size: 0
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http write filter: l:0 f:1 s:4096
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http write filter limit 2097152
2025/02/20 17:21:31 [debug] 1114#1114: *101569 writev: 4096 of 4096
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http write filter 0000000000000000
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http copy filter: 0 "/?"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 recv: eof:0, avail:0
2025/02/20 17:21:31 [debug] 1114#1114: *101569 event timer add: 15: 60000:17774858
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http upstream request: "/?"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http upstream send request handler
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http upstream send request
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http upstream send request body
2025/02/20 17:21:31 [debug] 1114#1114: *101569 chain writer in: 0000FFFF80287B90
2025/02/20 17:21:31 [debug] 1114#1114: *101569 writev: 1920 of 1920
2025/02/20 17:21:31 [debug] 1114#1114: *101569 chain writer out: 0000000000000000
2025/02/20 17:21:31 [debug] 1114#1114: *101569 read: -1, 0000FFFF802C1690, 8192, 368640
2025/02/20 17:21:31 [crit] 1114#1114: *101569 pread() "/tmp/nginx/client-body/0000000002" failed (9: Bad file descriptor) while sending request to upstream, client: 127.0.0.1, server: _, request: "POST / HTTP/1.1", upstream: "http://10.244.0.8:8080/", host: "localhost:8080"
2025/02/20 17:21:31 [debug] 1114#1114: *101569 http next upstream, 2
2025/02/20 17:21:31 [debug] 1114#1114: *101569 free keepalive peer
2025/02/20 17:21:31 [debug] 1114#1114: *101569 lua balancer free peer, tries: 2
2025/02/20 17:21:31 [debug] 1114#1114: *101569 finalize http upstream request: 502
2025/02/20 17:21:31 [debug] 1114#1114: *101569 finalize http proxy request

So, 0000FFFF802C1690 - is the buffer for /tmp/nginx/client-body/0000000002
Last read was at pos 360448, and then it tries to continue reads from pos 360448+8192=368640 to same buffer but from fh=-1?

Anything else we need to know:

We noticed the issue in both prod GKE and EKS clusters with ingress-nginx v1.11.2.
Tested that v1.12.0 and latest image with nginx 1.27 still have the issue.
The only found current workaround is to add:
nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
But that slows down all uploads to 2Mbit/s comparing to usual >300Mbit/s with buffering enabled, and is not acceptable.

Possibly related:
#4474
openresty/lua-nginx-module#2374

Also tried, without success:

• reduce nginx workers to 1
• bumping client_body_buffer_size can make 10Mb file pass, but then 11Mb fails
• client_body_in_file_only on;
• all variants of chunked_transfer_encoding, proxy_buffering

How can I help fixing this issue? Do you have more ideas to try?

[longwuyuan]

Can you grep your app code and confirm that you are not setting any chunking headers ?

[longwuyuan]

• FYI, this is not reproduced on a kind cluster for me ;

• I used my own upload server https://github.com/longwuyuan/file-server-and-client/blob/main/Dockerfile

• I added the directive https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size manually in the nginx, inside the image and set it to 14M

• I set the controller configMap option https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-body-size

• Without the configMap option and without the manual setting in the nginx inside the image, I got a 413 response as expected, because default size is 1M

• I created a 12M file uisng dd, to use for uploading

% k get po
NAME                                   READY   STATUS    RESTARTS   AGE
file-storage-server-6df985cd9b-m7wwr   1/1     Running   0          124m
[~] 
% k describe po file-storage-server-6df985cd9b-m7wwr| grep -i image:
    Image:          docker.io/library/file-storage-server
[~] 
% k describe svc file-storage-server 
Name:                     file-storage-server
Namespace:                default
Labels:                   app=file-storage-server
Annotations:              <none>
Selector:                 app=file-storage-server
Type:                     ClusterIP
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.109.63
IPs:                      10.96.109.63
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
Endpoints:                10.244.0.17:80
Session Affinity:         None
Internal Traffic Policy:  Cluster
Events:                   <none>
[~] 
% k describe ingress file-storage-server 
Name:             file-storage-server
Labels:           <none>
Namespace:        default
Address:          172.18.0.2
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  fs.dev.enjoydevops.com  
                          /   file-storage-server:80 (10.244.0.17:80)
Annotations:              nginx.ingress.kubernetes.io/client-body-buffer-size: 16K
Events:
  Type    Reason  Age                 From                      Message
  ----    ------  ----                ----                      -------
  Normal  Sync    89m (x8 over 123m)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    78m (x2 over 87m)   nginx-ingress-controller  Scheduled for sync

% k -n ingress-nginx describe cm ingress-nginx-controller 
Name:         ingress-nginx-controller
Namespace:    ingress-nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.12.0
              helm.sh/chart=ingress-nginx-4.12.0
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx

Data
====
proxy-body-size:
----
14M


BinaryData
====


% ls -lth testfile 
-rw-rw-r-- 1 me me 12M Feb 21 10:09 testfile

% curl --resolve fs.dev.enjoydevops.com:80:172.18.0.2 fs.dev.enjoydevops.com/upload -X POST \ 
  -H "Content-Type: multipart/form-data" \
  --form file="@testfile" -vv

Note: Unnecessary use of -X or --request, POST is already inferred.
* Added fs.dev.enjoydevops.com:80:172.18.0.2 to DNS cache
* Hostname fs.dev.enjoydevops.com was found in DNS cache
*   Trying 172.18.0.2:80...
* Connected to fs.dev.enjoydevops.com (172.18.0.2) port 80 (#0)
> POST /upload HTTP/1.1
> Host: fs.dev.enjoydevops.com
> User-Agent: curl/7.81.0
> Accept: */*
> Content-Length: 12583112
> Content-Type: multipart/form-data; boundary=------------------------d839ddb24bf4103f
> Expect: 100-continue
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 21 Feb 2025 06:51:30 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 20
< Connection: keep-alive
< 
* Connection #0 to host fs.dev.enjoydevops.com left intact
'testfile' uploaded!%                                             

- logs
2025/02/21 06:51:29 [warn] 1758#1758: *175599 a client request body is buffered to a temporary file /tmp/nginx/client-body/0000000010, client: 172.18.0.1, server: fs.dev.enjoydevops.com, request: "POST /upload HTTP/1.1", host: "fs.dev.enjoydevops.com"
172.18.0.1 - - [21/Feb/2025:06:51:30 +0000] "POST /upload HTTP/1.1" 200 45 "-" "curl/7.81.0" 12583339 0.403 [default-file-storage-server-80] [] 10.244.0.17:80 20 0.384 200 c3ea39e84dc5a9d06b47c8c7ca768ded

[longwuyuan]

You deployed a beta release of the controller. Dont use that.
Use the latest release and use the official repo and not the kind example.

Since it can not be reproduced at will, let us wait for traige data to prove the bug, before we apply the bug label here

/remove-kind bug
/kind support
/triage needs-information

[longwuyuan]

In the description, please remove the raw yaml destined to be applied. Instead post the real live state of the object using kubectl describe or kubectl get ... -o yaml

[sepich]

confirm that you are not setting any chunking headers

Could you please provide more info on why setting Transfer-Encoding: chunked is not supported by ingress-nginx?
Per debug logs above, the test app sets this header only once in the response. Per:
https://kubernetes.github.io/ingress-nginx/faq/#why-is-chunking-not-working-since-controller-v110
I do not see upstream sent duplicate header errors in this case.

I used my own upload server
Since it can not be reproduced at will

I'm sorry you wasted time with your own upload server. Now please follow my reproduction steps exactly, for that you need to find "yaml spec" section in the original message, and click on it.
If you think that issue is related to upstream response, then how any rubbish response could lead to Bad file descriptor inside nginx? And how the same request starts to work fine without any changes to upstream but only disabling request_buffering on nginx side?

You deployed a beta release of the controller

Version does not matter, as I've said:

We noticed the issue in both prod GKE and EKS clusters with ingress-nginx v1.11.2.
Tested that v1.12.0 and latest image with nginx 1.27 still have the issue.

But ok, lets fix the instruction to have:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml

post the real live state of the object

Please find them clicking below:

Current State of the controller

$ kubectl describe ingressclasses
Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.12.0
Annotations:  <none>
Controller:   k8s.io/ingress-nginx
Events:       <none>

$ kubectl -n ingress-nginx get all  -o wide
NAME                                            READY   STATUS      RESTARTS   AGE    IP            NODE                 NOMINATED NODE   READINESS GATES
pod/ingress-nginx-admission-create-mwgdj        0/1     Completed   0          2m6s   10.244.0.9    kind-control-plane   <none>           <none>
pod/ingress-nginx-admission-patch-zwx5d         0/1     Completed   1          2m6s   10.244.0.10   kind-control-plane   <none>           <none>
pod/ingress-nginx-controller-6d59d95796-p2wkv   1/1     Running     0          2m6s   10.244.0.11   kind-control-plane   <none>           <none>

NAME                                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE    SELECTOR
service/ingress-nginx-controller             NodePort    10.96.220.120   <none>        80:32105/TCP,443:32050/TCP   2m6s   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-admission   ClusterIP   10.96.87.92     <none>        443/TCP                      2m6s   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES                                                                                                                     SELECTOR
deployment.apps/ingress-nginx-controller   1/1     1            1           2m6s   controller   registry.k8s.io/ingress-nginx/controller:v1.12.0@sha256:e6b8de175acda6ca913891f0f727bca4527e797d52688cbe9fec9040d6f6b6fa   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME                                                  DESIRED   CURRENT   READY   AGE    CONTAINERS   IMAGES                                                                                                                     SELECTOR
replicaset.apps/ingress-nginx-controller-6d59d95796   1         1         1       2m6s   controller   registry.k8s.io/ingress-nginx/controller:v1.12.0@sha256:e6b8de175acda6ca913891f0f727bca4527e797d52688cbe9fec9040d6f6b6fa   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6d59d95796

NAME                                       STATUS     COMPLETIONS   DURATION   AGE    CONTAINERS   IMAGES                                                                                                                              SELECTOR
job.batch/ingress-nginx-admission-create   Complete   1/1           7s         2m6s   create       registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.0@sha256:aaafd456bda110628b2d4ca6296f38731a3aaf0bf7581efae824a41c770a8fc4   batch.kubernetes.io/controller-uid=e4c44efb-4b94-4b4b-a96c-224712888065
job.batch/ingress-nginx-admission-patch    Complete   1/1           8s         2m6s   patch        registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.0@sha256:aaafd456bda110628b2d4ca6296f38731a3aaf0bf7581efae824a41c770a8fc4   batch.kubernetes.io/controller-uid=df9e93ea-4f8a-46a8-bba2-267fee933b9b

$ kubectl -n ingress-nginx describe pod/ingress-nginx-controller-6d59d95796-p2wkv
Name:             ingress-nginx-controller-6d59d95796-p2wkv
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             kind-control-plane/192.168.224.2
Start Time:       Fri, 21 Feb 2025 09:06:47 +0100
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.12.0
                  pod-template-hash=6d59d95796
Annotations:      <none>
Status:           Running
IP:               10.244.0.11
IPs:
  IP:           10.244.0.11
Controlled By:  ReplicaSet/ingress-nginx-controller-6d59d95796
Containers:
  controller:
    Container ID:    containerd://b92df74f4b7ba33ececc2c3e9365bbf3e793ee409ec27208b0acfecb07b513ab
    Image:           registry.k8s.io/ingress-nginx/controller:v1.12.0@sha256:e6b8de175acda6ca913891f0f727bca4527e797d52688cbe9fec9040d6f6b6fa
    Image ID:        registry.k8s.io/ingress-nginx/controller@sha256:e6b8de175acda6ca913891f0f727bca4527e797d52688cbe9fec9040d6f6b6fa
    Ports:           80/TCP, 443/TCP, 8443/TCP
    Host Ports:      0/TCP, 0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      /nginx-ingress-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Fri, 21 Feb 2025 09:07:04 +0100
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-6d59d95796-p2wkv (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2r9c9 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-2r9c9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                From                      Message
  ----     ------       ----               ----                      -------
  Normal   Scheduled    55m                default-scheduler         Successfully assigned ingress-nginx/ingress-nginx-controller-6d59d95796-p2wkv to kind-control-plane
  Warning  FailedMount  55m (x4 over 55m)  kubelet                   MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
  Normal   Pulling      55m                kubelet                   Pulling image "registry.k8s.io/ingress-nginx/controller:v1.12.0@sha256:e6b8de175acda6ca913891f0f727bca4527e797d52688cbe9fec9040d6f6b6fa"
  Normal   Pulled       54m                kubelet                   Successfully pulled image "registry.k8s.io/ingress-nginx/controller:v1.12.0@sha256:e6b8de175acda6ca913891f0f727bca4527e797d52688cbe9fec9040d6f6b6fa" in 9.668s (9.668s including waiting). Image size: 103810349 bytes.
  Normal   Created      54m                kubelet                   Created container: controller
  Normal   Started      54m                kubelet                   Started container controller
  Normal   RELOAD       54m                nginx-ingress-controller  NGINX reload triggered due to a change in configuration

$ kubectl -n ingress-nginx describe svc/ingress-nginx-controller
Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.12.0
Annotations:              <none>
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.220.120
IPs:                      10.96.220.120
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  32105/TCP
Endpoints:                10.244.0.11:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  32050/TCP
Endpoints:                10.244.0.11:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

$ k describe ing echo
Name:             echo
Labels:           <none>
Namespace:        kube-system
Address:          192.168.224.2
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *
              /   echo:8080 (10.244.0.8:8080)
Annotations:  nginx.ingress.kubernetes.io/proxy-body-size: 0
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    54m (x3 over 55m)  nginx-ingress-controller  Scheduled for sync

$ k -n ingress-nginx logs --tail=5 ingress-nginx-controller-6d59d95796-p2wkv
I0221 08:08:06.312940      11 status.go:304] "updating Ingress status" namespace="kube-system" ingress="echo" currentValue=null newValue=[{"ip":"192.168.224.2"}]
I0221 08:08:06.317701      11 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kube-system", Name:"echo", UID:"eb48935e-1c17-40d5-9563-599b29c8d844", APIVersion:"networking.k8s.io/v1", ResourceVersion:"41523", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
2025/02/21 09:03:38 [warn] 38#38: *49553 a client request body is buffered to a temporary file /tmp/nginx/client-body/0000000001, client: 127.0.0.1, server: _, request: "POST / HTTP/1.1", host: "localhost:8080"
2025/02/21 09:03:38 [crit] 38#38: *49553 pread() "/tmp/nginx/client-body/0000000001" failed (9: Bad file descriptor) while sending request to upstream, client: 127.0.0.1, server: _, request: "POST / HTTP/1.1", upstream: "http://10.244.0.8:8080/", host: "localhost:8080"
127.0.0.1 - - [21/Feb/2025:09:03:39 +0000] "POST / HTTP/1.1" 200 3959 "-" "curl/8.7.1" 10363455 1.034 [kube-system-echo-8080] [] 10.244.0.8:8080 3939 0.003 502 405ed6177f17fd9279ff2e0118681877

Hope that helps.

[sepich]

BTW, I've tested ingress-nginx v1.9.4 (nginx v1.21.6) and it has the same issue.
So it is not related to #11162 (chunking header dup = 502)

[longwuyuan]

@sepich thanks for the details. it does help in some ways.

• The chunking is a upstream Nginx fact that its forced on out of the box. There was a issue we worked on earlier where we found that regardless of any factor, setting the chunking header in the app code does not integrate with the Nginx bundled in the controller. That user had to remove the lines of code that set the chunking header in their app. I understand you can reproduce the problem in an older version of the controller and there is no chunking related error message logged. So chunking may be a red-herring. Lets address it later

• Since you wanted me to reproduce using your echo server, I will try that. But before I try that, I need to know why I can not use any upload-server-app. The ingress-nginx controller will work the same with any backend app that has a upload api.

• Are you claiming that request buffering is one major factor here to reproduce the bad-descriptor. As far as I understand, I should have faced the same bad-descriptor error. You set the proxy-buffer-size annotation to 0 and I set it it to 14M globally (via the configMap) and uploaded 12M file.