v 2.x No Readiness or Liveness Probes for the Data Plane

[vazkarvishal]

Describe the bug
In the previous versions of NGF, the pod had a readiness probe on port 8081. This readiness probe has carried over to the gateway pod/deployment, but is missing from the dataplane pod. Thus, the data plane pod is deployed without any health checks.

Based on the Proposal 1508 Liveness & Readiness Probes.
Control plane should not restart data plane pods if they are unhealthy. This can either be left in the hands of the users, or if utilizing a liveness probe, Kubernetes will restart the pod. However, I am unable to (as the user) control these probes as they dont seem to be configurable.

To Reproduce
Steps to reproduce the behavior:

1. Deploy 2.x and inspect the dataplane to look out for health checks.

Expected behavior
A Readiness probe should be available to accurately test whether the dataplane is ready to serve traffic.

Your environment

• v2.0.1

[sjberman]

@vazkarvishal Do you plan on mounting a custom nginx config that has an endpoint for health checking? Just curious how you would define the probe endpoint.

We don't create any default health endpoints in nginx, since all endpoints are purely based on a user's Gateway and Route resources. I did create #3568 to handle more use cases for users to set Deployment/Service configuration that we haven't covered yet in our NginxProxy CRD, currently targeted at our 2.2 release.

[vazkarvishal]

@sjberman not really, but the challenge we have here is that there is a Loadbalancer sending traffic into the data plane directly. The health checks were performed using TCP on the health check endpoints which were available in the 1.x version. Not having a health endpoint means the Loadbalancer can send traffic to a potentially unhealthy container in a loadbalanced environment, because the container may be "TCP" ready but not "Http" ready yet for whatever reasons?

[sjberman]

We could expose a way for users to define these probes in their Deployment config, but I'm trying to think how we would actually define a health check server in nginx. By default when we start up nginx, there are no health check servers running. If we did define it, we would need to ensure that it won't collide with any user configuration (via their Gateway/Routes) and probably allow for configuring it as well.

A workaround in the meantime could be implementing #3568 to allow users to patch in their probes into the config, and use their HTTPRoute endpoint as the health check server.

[vazkarvishal]

potentially. However, would it be worth adding a sensible default with an override? I am thinking out loud here, any K8s cluster configured with best practice Kyverno policies or something similar may block these deployments (like they are for me, but I have got around it using warn mode for now)

https://kyverno.io/policies/best-practices/require-probes/require-probes/

[sjberman]

I'll chat with the team more about this to see if we can build in a sensible default. Good to know that some environments require this.

[vazkarvishal]

Thanks, even a default healthz endpoint like nginx ingress controller should suffice I think.