Multiple ingress classes (f. ex internal / external nginx)

[mattia-crypto]

EDIT
This is now focused on multiple ingress class names.

OLD Version

Out of curiosity, is there a working configuration example shown to work for the Kafka CRD which uses:

• ExternalDNS
• CertManager
• listener[].configuration.class: ingress

Below you will see my listener configurations. However, they give very strange results which I think verifies a few questions I have seen around this repository around tls in the listener configuration blocks... First and foremost, it confirms that only nginx ingress controller works.

ALB Controller in AWS:
Note that I created the kafka-tls-certs-external using cert-manager CRDs.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
  namespace: kafka
spec:
  kafka:
    version: 3.1.0
    replicas: 3
    # image: ""
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
        authentication:
          type: scram-sha-512
      - name: external
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: tls
        configuration:
          class: alb
          brokerCertChainAndKey:
            secretName: kafka-tls-certs-external
            certificate: tls.crt
            key: tls.key
          bootstrap:
            host: bootstrap.example.dev
            annotations:
              external-dns.alpha.kubernetes.io/hostname: bootstrap.example.dev
              external-dns.alpha.kubernetes.io/ttl: "60"
          brokers:
            - broker: 0
              host: kafka-broker-0.example.dev
              annotations:
                external-dns.alpha.kubernetes.io/hostname: kafka-broker-0.example.dev
                external-dns.alpha.kubernetes.io/ttl: "60"
            - broker: 1
              host: kafka-broker-1.example.dev
              annotations:
                external-dns.alpha.kubernetes.io/hostname: kafka-broker-1.example.dev
                external-dns.alpha.kubernetes.io/ttl: "60"
            - broker: 2
              host: kafka-broker-2.example.dev
              annotations:
                external-dns.alpha.kubernetes.io/hostname: kafka-broker-2.example.dev
                external-dns.alpha.kubernetes.io/ttl: "60"

You can see that in this case I simply use the Ingress capabilities of ALB. However, this does not create any Ingress resources in my release namespace. Neither does it create Service of type LoadBalancer, it simply creates a Service of type ClusterIP on a per N + 1 brokers (+1 for bootstrap) :

NAME                                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
kafka-cluster-kafka-0                    ClusterIP   192.168.3.184   <none>        9094/TCP                     64s
kafka-cluster-kafka-1                    ClusterIP   192.168.5.134   <none>        9094/TCP                     64s
kafka-cluster-kafka-2                    ClusterIP   192.168.6.196   <none>        9094/TCP                     64s
kafka-cluster-kafka-bootstrap            ClusterIP   192.168.13.80   <none>        9091/TCP,9092/TCP            64s
kafka-cluster-kafka-brokers              ClusterIP   None            <none>        9090/TCP,9091/TCP,9092/TCP   64s
kafka-cluster-kafka-external-bootstrap   ClusterIP   192.168.0.139   <none>        9094/TCP                     64s
kafka-cluster-zookeeper-client           ClusterIP   192.168.7.138   <none>        2181/TCP                     104s
kafka-cluster-zookeeper-nodes            ClusterIP   None            <none>        2181/TCP,2888/TCP,3888/TCP   104s

My ALB logs indicate that nothing is happening on the ALB Controller side. Additionally, no brokers are being deployed, and after checking the strimzi-operator logs, I see this section, which is crucial:

Message: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-internal-controller-admission.cluster-system.svc:443/networking/v1/ingresses?timeout=10s": no endpoints available for service "ingress-nginx-internal-controller-admission"

It proves that only the nginx-ingress-controller can be used. It makes sense that this does not trigger deployment of any brokers accordingly.

Naturally, I would try with the nginx-ingress-controller. The change is very simple, as I effectively only need to change the listener[].configuration.class parameter.

NGINX Ingress Controller listener config:
Note that I created the kafka-tls-certs-external using cert-manager CRDs here as well. Regardless, I have deployed an ingress-nginx controller which is defined with an ingressClass of nginx-external and that works perfectly fine.

listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
        authentication:
          type: scram-sha-512
      - name: external
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: tls
        configuration:
          class: nginx-external
          brokerCertChainAndKey:
            secretName: kafka-tls-certs-external
            certificate: tls.crt
            key: tls.key
          bootstrap:
            host: bootstrap.example.dev
            annotations:
              external-dns.alpha.kubernetes.io/hostname: bootstrap.example.dev
              external-dns.alpha.kubernetes.io/ttl: "60"
          brokers:
            - broker: 0
              host: kafka-broker-0.example.dev
              annotations:
                external-dns.alpha.kubernetes.io/hostname: kafka-broker-0.example.dev
                external-dns.alpha.kubernetes.io/ttl: "60"
            - broker: 1
              host: kafka-broker-1.example.dev
              annotations:
                external-dns.alpha.kubernetes.io/hostname: kafka-broker-1.example.dev
                external-dns.alpha.kubernetes.io/ttl: "60"
            - broker: 2
              host: kafka-broker-2.example.dev
              annotations:
                external-dns.alpha.kubernetes.io/hostname: kafka-broker-2.example.dev
                external-dns.alpha.kubernetes.io/ttl: "60"

This time I got this error in the operator logs:

2022-03-02 18:08:34 INFO  Main:151 - Cluster Operator verticle started in namespace kafka without label selector
2022-03-02 18:08:37 INFO  OperatorWatcher:38 - Reconciliation #1(watch) Kafka(kafka/kafka-cluster): Kafka kafka-cluster in namespace kafka was ADDED
2022-03-02 18:08:37 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkas' with unstable version 'v1beta2'
2022-03-02 18:08:37 INFO  AbstractOperator:226 - Reconciliation #1(watch) Kafka(kafka/kafka-cluster): Kafka kafka-cluster will be checked for creation or modification
2022-03-02 18:08:37 INFO  OperatorWatcher:38 - Reconciliation #2(watch) Kafka(kafka/kafka-cluster): Kafka kafka-cluster in namespace kafka was MODIFIED
2022-03-02 18:08:37 INFO  CrdOperator:113 - Reconciliation #1(watch) Kafka(kafka/kafka-cluster): Status of Kafka kafka-cluster in namespace kafka has been updated
2022-03-02 18:08:41 WARN  VersionUsageUtils:60 - The client is using resource type 'strimzipodsets' with unstable version 'v1beta2'
2022-03-02 18:08:42 WARN  VersionUsageUtils:60 - The client is using resource type 'poddisruptionbudgets' with unstable version 'v1beta1'
2022-03-02 18:09:37 INFO  AbstractOperator:373 - Reconciliation #1(watch) Kafka(kafka/kafka-cluster): Reconciliation is in progress
2022-03-02 18:09:41 ERROR AbstractOperator:247 - Reconciliation #1(watch) Kafka(kafka/kafka-cluster): createOrUpdate failed
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:103) ~[io.fabric8.kubernetes-client-5.12.0.jar:?]
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:97) ~[io.fabric8.kubernetes-client-5.12.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:63) ~[io.fabric8.kubernetes-client-5.12.0.jar:?]
	at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.internalCreate(AbstractResourceOperator.java:275) ~[io.strimzi.operator-common-0.28.0.jar:0.28.0]
	at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.lambda$reconcile$0(AbstractResourceOperator.java:113) ~[io.strimzi.operator-common-0.28.0.jar:0.28.0]
	at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]

Also:

kafka-cluster-kafka-0                    ClusterIP   192.168.2.73     <none>        9094/TCP                     96s
kafka-cluster-kafka-1                    ClusterIP   192.168.11.178   <none>        9094/TCP                     96s
kafka-cluster-kafka-2                    ClusterIP   192.168.13.237   <none>        9094/TCP                     96s
kafka-cluster-kafka-bootstrap            ClusterIP   192.168.11.235   <none>        9091/TCP,9092/TCP            96s
kafka-cluster-kafka-brokers              ClusterIP   None             <none>        9090/TCP,9091/TCP,9092/TCP   96s
kafka-cluster-kafka-external-bootstrap   ClusterIP   192.168.6.197    <none>        9094/TCP                     96s
kafka-cluster-zookeeper-client           ClusterIP   192.168.3.117    <none>        2181/TCP                     2m25s
kafka-cluster-zookeeper-nodes            ClusterIP   None             <none>        2181/TCP,2888/TCP,3888/TCP   2m25s

However, no ingress resources were made... I am really not sure what could be going wrong here.

[scholzj]

You can check this blog post - https://strimzi.io/blog/2021/05/07/deploying-kafka-with-lets-encrypt-certificates/ - which is very similar. The main difference is that Strimzi is designed, developed and tested with Kubernetes Nginx Controller. Not sure if your ALB controller will work. I assume ALB is Application Loadb Balancer? Does it support TLS passthrough on which Strimzi depends?

In any case, if it didn't created the Ingress resources, it probably failed before that. I don't think I ever saw an exception like this which is without any proper reason. Maybe if you change the log level to DEBUG, it would show at least what were the last things it was doing.

[mattia-crypto]

Indeed it is very strange which is why I wanted to bring it up in this discussion

[mattia-crypto]

And yes almost :-) it is the AWS Load Balancer Controller.

[mattia-crypto]

Will add some more logs in here

[mattia-crypto]

It does seem that Strimzi gets confused if there are multiple ValidatingWebhookConfigurations for the ingress-nginx-controller. Some people use multiple ingress-nginx-controllers for example, when you need an internal and external ingress class.

I managed to get the external configuration working by deleting the internal ValidatingWebhookConfiguration. I shouldn't really need to do this IMO if the internal ingress controller is not deployed, which it wasn't.

[mattia-crypto]

Also, this error message from the operator logs does indicate that the admission webhook is triggered by it somehow, and yes you are right that the admission controller somehow protects what gets created in the cluster.

Message: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-internal-controller-admission.cluster-system.svc:443/networking/v1/ingresses?timeout=10s": no endpoints available for service "ingress-nginx-internal-controller-admission"

[scholzj]

Also, this error message from the operator logs does indicate that the admission webhook is triggered by it somehow, and yes you are right that the admission controller somehow protects what gets created in the cluster.

No, this is the error message we get from Kubernetes as a response when trying to create the resource. It tells us why the resource cannot be created.

It is also worth mentioning that it is failing because the webhook is not running and not because it failed. So it is not clear if it would actually fail it the webhook was running.

Or perhaps you have an example of how you would configure the nginx-ingress-controller so that the admissionWebhook works for the different listeners in the same Kafka resource?

More precisely put, how can I customise the admissionWebhook to target different listener configurations in the Kafka resource? In this case it seems the only option is to disable the admissionWebhook:

TBH, this is a question on Kubernetes Nginx Controller project. It is their webhook, not ours. So it is their responsibility. Also you do not even have to use the webhook, it is just some optional validation AFAIK.

In this case, the Kafka resource is a single resource given that is a CRD. This is effectively why I raised this discussion. Is there no way to use multiple ingress classes in Strimzi for the listener? Perhaps there is an example already.

The Ingress resource supports the ingressClassName field to distinguish between different Ingress controllers. So you can create multiple listeners, each using different Ingress class using https://strimzi.io/docs/operators/latest/full/configuring.html#property-listener-config-class-reference ... and based on the class, each Ingress controller picks up the right set of Ingress resources fur each listener. So this works fine as AFAIK there are users using it.

[mattia-crypto]

I mean even if it was running, the ingressClassName I am specifying is pointing to a different controller....

[scholzj]

Well, as I said, this is something what you probably need to consult the Kubernetes Nginx Ingress community about. It is their Validation webhook, so they should explain how they expect it to work in that case.

[mattia-crypto]

It seems the only option is to disable the webhook.