How to implement graceful shutdown with the controller?

[rukeith-paypay]

What happened:

I deployed the Ingress Nginx controller to AWS EKS. I tried to access the pod and send the nginx -s quit command. But I cannot find the graceful shutdown log from the standard output.

What you expected to happen:
I expected the graceful shutdown log will show up at standard output like this.

NGINX Ingress controller version

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       1.21.1
  Build:         64780b1
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.5

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: v1.28.9
Kustomize Version: v5.0.4.0.20230601165947-6ce0bf390ce3
Server Version: v1.29.15-eks-4096722

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[rblaine95]

In our AWS EKS cluster, we use the following to achieve graceful shutdown of our Ingress Nginx pods:

controller:
  replicaCount: 2
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
  terminationGracePeriodSeconds: 300
  lifecycle:
    preStop:
      exec:
        command:
          - /bin/sh
          - -c
          - |
            sleep 15
            /usr/local/nginx/sbin/nginx -c /etc/nginx/nginx.conf -s quit
            while pgrep -f '^nginx: (worker|master|cache manager) process'; do
              echo "waiting for nginx to gracefully shutdown"
              sleep 1
            done

Edit: Explanations of why this config
replicaCount: 2 - High availability
updateStrategy.rollingUpdate.maxUnavailable: 1 - only roll 1 pod at a time during an update
terminationGracePeriodSeconds: 300 - after 5 minutes of Terminating, k8s will forcefully kill the pod - something most likely went wrong - we've never actually seen this trigger

The preStop lifecycle:
sleep 15 - When the pod goes into a Terminating state, the AWS A/NLB begins draining active connections to the pod. This allows the preStop hook to catch and hold onto the SIGTERM, giving the AWS A/NLB enough time to register the Terminating state and begin draining connections.
nginx -c /etc/nginx/nginx.conf -s quit - send SIGQUIT to nginx
while pgrep -f ... - print to stdout that we are waiting for nginx to gracefully shutdown, sleep for 1 second, check if anything is still running