Ingress-nginx pod crashes due to illegal instruction on arm64 version 1.13.0 (helm 4.13.0)

[LarssonOliver]

What happened:

Hi 👋

Recently upgraded to helm chart version 4.13.0 and thus controller version 1.13.0 across my clusters.

I have both amd64 and arm64 clusters. The amd64 ones work as expected, but the arm64 deployment quickly crashes after initial startup:

I0711 07:08:41.110797       7 controller.go:203] "Configuration changes detected, backend reload required"
I0711 07:08:41.121628       7 status.go:85] "New leader elected" identity="ingress-nginx-controller-867db7f689-g5882"
W0711 07:08:41.228438       7 nginx.go:37]
-------------------------------------------------------------------------------
NGINX master process died (-1): signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
E0711 07:08:41.528826       7 controller.go:218] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
E0711 07:08:41.529029       7 queue.go:131] "requeuing" err=<
	-------------------------------------------------------------------------------
	Error: signal: illegal instruction (core dumped)
	-------------------------------------------------------------------------------
 > key="initial-sync"
I0711 07:08:41.529190       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-67b664d98f-zsr29", UID:"8b4e924b-7138-460f-ab54-1451e7730ac8", APIVersion:"v1", ResourceVersion:"184201005", FieldPath:""}): type: 'Warning' reason: 'RELOAD' Error reloading NGINX:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
I0711 07:08:44.443849       7 controller.go:203] "Configuration changes detected, backend reload required"
E0711 07:08:44.676647       7 controller.go:218] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------

For reference:

E0711 07:08:37.800130       7 main.go:157] "unexpected error obtaining NGINX version" err="signal: illegal instruction (core dumped)"
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.13.0
  Build:         4cbb78a9dc4f1888af802b70ddf980272e01268b
  Repository:    https://github.com/kubernetes/ingress-nginx
  N/A
-------------------------------------------------------------------------------
W0711 07:08:37.828055       7 client_config.go:667] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.

Any idea what could be causing this? This happened on a Raspberry Pi 4.

What you expected to happen:

The pod should start and run without crashing.

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):

Pod crashes too fast :/

Kubernetes version (use kubectl version):

Client Version: v1.32.2
Kustomize Version: v5.5.0
Server Version: v1.32.3+k3s1

Environment:

• Cloud provider or hardware configuration:
  Raspberry Pi 4

• OS (e.g. from /etc/os-release):
  Debian GNU/Linux 11 (bullseye)

• Kernel (e.g. uname -a):
  Linux hostname 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux

• How was the ingress-nginx-controller installed:
  Helm chart managed by ArgoCD

How to reproduce this issue:

Anything else we need to know:

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[strongjz]

To my knowledge our e2e testing is only done on amd64, so we apologize for this issue. We will look into it after we fix this issue #13598

[Gacko]

Can you please try the same with v1.12.4? I tested it locally on my M1 MacBook, which is arm64, and it works. I'm wondering if this is connected to the NGINX update.

[LarssonOliver]

Can confirm the issue occurs with v1.12.4, both with chart versions 4.13.0 and 4.12.3.

I might be wrong here, but I think the M1 silicon implements the ARMv8.4-A instruction set, whereas this raspberry pi 4 implements the ARMv8.0-A set. Could there be some extended instruction(s?) added somewhere between the 8.0 and 8.4 versions at play here?

Logs from this one:

E0723 06:34:32.352959       7 main.go:157] "unexpected error obtaining NGINX version" err="signal: illegal instruction (core dumped)"
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.12.4
  Build:         347ffcba3e27350a5d15505376fe99a72ca6d504
  Repository:    https://github.com/kubernetes/ingress-nginx
  N/A
-------------------------------------------------------------------------------
W0723 06:34:32.392306       7 client_config.go:667] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0723 06:34:32.396407       7 main.go:205] "Creating API client" host="https://10.43.0.1:443"
I0723 06:34:32.472816       7 main.go:248] "Running in Kubernetes cluster" major="1" minor="32" git="v1.32.3+k3s1" state="clean" commit="079ffa8d99fb859cb8c001455e47efa65535d832" platform="linux/arm64"
I0723 06:34:33.180626       7 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0723 06:34:33.285015       7 ssl.go:535] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0723 06:34:33.423232       7 nginx.go:271] "Starting NGINX Ingress controller"
I0723 06:34:33.483278       7 event.go:377] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"ef64dadf-bcc1-4048-87a0-6bcd6ee98c64", APIVersion:"v1", ResourceVersion:"191514218", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0723 06:34:33.611240       7 event.go:377] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-tcp", UID:"acd2dc5e-5de8-42a4-a9d3-7bb5c4fac0c4", APIVersion:"v1", ResourceVersion:"191514220", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-tcp
I0723 06:34:33.611339       7 event.go:377] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-udp", UID:"c7759077-d0c3-45b6-a740-86e4f587e558", APIVersion:"v1", ResourceVersion:"191514219", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-udp

<redacted discovery logs of ingress and secret objects>

I0723 06:34:34.626163       7 leaderelection.go:257] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader...
I0723 06:34:34.626053       7 nginx.go:317] "Starting NGINX process"
I0723 06:34:34.632080       7 nginx.go:337] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0723 06:34:34.636972       7 controller.go:196] "Configuration changes detected, backend reload required"
I0723 06:34:34.653105       7 status.go:85] "New leader elected" identity="ingress-nginx-controller-57fc8fbc7d-6k7mz"
W0723 06:34:34.782469       7 nginx.go:37]
-------------------------------------------------------------------------------
NGINX master process died (-1): signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
I0723 06:34:34.838262       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"nextcloud", Name:"contacts", UID:"552448b2-22dd-4513-9571-23ec2be67a42", APIVersion:"networking.k8s.io/v1", ResourceVersion:"191514396", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0723 06:34:34.955664       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"airtrail", Name:"airtrail", UID:"1a464e13-73e7-4f68-b30c-8671a74f3d86", APIVersion:"networking.k8s.io/v1", ResourceVersion:"191514399", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
E0723 06:34:35.026772       7 controller.go:211] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
E0723 06:34:35.027051       7 queue.go:131] "requeuing" err=<
	-------------------------------------------------------------------------------
	Error: signal: illegal instruction (core dumped)
	-------------------------------------------------------------------------------
 > key="initial-sync"
I0723 06:34:35.027229       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-856d89cdbc-v7r4j", UID:"da5bd720-460f-4527-a9ae-c6625e036d0c", APIVersion:"v1", ResourceVersion:"191514352", FieldPath:""}): type: 'Warning' reason: 'RELOAD' Error reloading NGINX:
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------

[Gacko]

The NGINX image is being built on the same infrastructure as the controller image (Google Cloud Build, see logs below). I did not build them locally, so we should run the same image. Also what's failing here is the NGINX binary, which is being called by the controller.

So we're using the same image built by the same infrastructure on different architectures and while the controller part works, the NGINX part fails.

These are the logs for building the NGINX base image, maybe this helps: https://storage.googleapis.com/kubernetes-ci-logs/logs/post-ingress-nginx-nginx/1938100646695669760/artifacts/build.log

Also thank you very much for also testing v1.12.4! Would you do me one more favor and also test v1.12.3? Because the diff between v1.12.3 and v1.12.4 is way smaller and easier to investigate than the diff between v1.12.3 and v1.13.0.

I would try this on my own, but it seems like this is really specific to your environment and I do not have a Raspberry Pi 4 lying around (anymore). 😔

[LarssonOliver]

Thanks for the quick reply! I'll have to look into this a little later, but I'll get back to you soon :)

[Gacko]

Ah and one more thing: You're telling you upgraded to v1.13.0. Which version did you upgrade from? Because if some versions older than v1.12.4/v1.13.0 also aren't working for you, we can somehow be sure this ain't a regression introduced in v1.12.4/v1.13.0.

[LarssonOliver]

I upgraded from chart version 4.12.3, controller version v1.12.3.

Can confirm the new chart version 4.13.0 works with controller version v1.12.3. Running that right now.

So this was introduced somewhere in the v1.12.3 - v1.12.4 diff.

[Gacko]

So just to confirm my understanding: The issue is somewhere in the controller / NGINX code changed between v1.12.3 and v1.12.4, because chart v4.13.0 with controller v1.12.3 works while controller v1.12.4 doesn't? And since the controller seems to come up fine but NGINX doesn't we can further narrow it down to just the NGINX image, at least in a first step?

controller-v1.12.3...controller-v1.12.4