Kafka external load balancer support on port 443 #7466

rajjaiswalsaumya · 2022-10-12T18:56:32Z

rajjaiswalsaumya
Oct 12, 2022

We want to expose Kafka over ingress and route from 443 to kafka. We tried ingress but that does not seem to work for us. Can you please provide some sample tutorial to deploy ingress controller correcly that can work with strimzi ?

scholzj · 2022-10-12T18:59:22Z

scholzj
Oct 12, 2022
Maintainer

Strimzi is developed and tested with support for Kubernetes Nginx Ingress using the TLS passthrough. So you can follow their docs to install the controller and then Strimzi docs to deploy Kafka with Ingress listener.

12 replies

rajjaiswalsaumya Oct 17, 2022
Author

Hi @scholzj Apologies for not adding logs or details. Let me try to rephrase the full context. I am trying to expose Kafka via Ingress on GKE. As suggested in docs, I deployed my Nginx controller and enabled tls pass through using --enable-ssl-passthrough Then deployed my cluster using Strimzi Operator.

Here is my listener configuration

    listeners:
      ### plain : Listens on port 9092 (without TLS encryption) and supports "scram-sha-512" authentication
      plain:
        enabled: false
        # authentication:
        #   type: scram-sha-512
      ### tls : Listens on port 9093 (with TLS encryption)
      ### Supports "scram-sha-512" and "tls" authentication and its respective configurations
      ### The nested structed for configuring "authentication" and "configuration" can be referred from the README/official Strimzi document
      tls:
        enabled: true
        authentication:
          type: tls
      ### external : Listens on port 9094 for access from outside of Kubernetes
      external:
        enabled: true
        ###loadbalancer,nodeport,ingress
        type: ingress
        port: 9094
        tls: true
        configuration:
          externalTrafficPolicy: Local
          bootstrap:
            host: kafka-ccdf-2-kafka-bootstrap.sedjetstream-ccdf-dev-kafka-ccdf-2.svc
            annotations:
              kubernetes.io/ingress.class: sedjetstream-ccdf-dev-nginx
          brokers:
            - broker: 0
              host: kafka-ccdf-2-kafka-0.sedjetstream-ccdf-dev-kafka-ccdf-2.svc
              annotations:
                kubernetes.io/ingress.class: sedjetstream-ccdf-dev-nginx
            - broker: 1
              host: kafka-ccdf-2-kafka-1.sedjetstream-ccdf-dev-kafka-ccdf-2.svc
              annotations:
                kubernetes.io/ingress.class: sedjetstream-ccdf-dev-nginx
            - broker: 2
              host: kafka-ccdf-2-kafka-2.sedjetstream-ccdf-dev-kafka-ccdf-2.svc
              annotations:
                kubernetes.io/ingress.class: sedjetstream-ccdf-dev-nginx
    ### Kafka authorization and its respective attributes
    authorization:
      type: simple
    ### The template can be used to configure details such as externalBootstrapService, perPodService etc.,
    ### Additional template properties can be applied under the "spec" attribute which can be can be referred from the README/official Strimzi document
    template:
      enabled: false
      spec:
        externalBootstrapService:
          metadata:
            annotations:
              networking.gke.io/load-balancer-type: "Internal"
              networking.gke.io/internal-load-balancer-allow-global-access: "true"
        perPodService:
          metadata:
            annotations:
              networking.gke.io/load-balancer-type: "Internal"
              networking.gke.io/internal-load-balancer-allow-global-access: "true"

I can see Ingress deployed in GKE but that is not getting IP Address. And when I try to connect using kafka topic script using console using properties and user certs, I am getting tls error.

Kafka properties used on console

ssl.truststore.type=PKCS12
ssl.keystore.type=PKCS12
security.protocol=SSL
ssl.truststore.location=/Users/rg735009/Development/workspaces/kafka/secrets/DEV2/certs/ca.p12
ssl.keystore.location=/Users/rg735009/Development/workspaces/kafka/secrets/DEV2/certs/user.p12
ssl.truststore.password=abcdf344feiji
ssl.keystore.password=iwnwch28484

Contents of cert dir

ls /Users/rg735009/Development/workspaces/kafka/secrets/DEV2/certs
ca.p12 user.p12

Command used on Mac to run kafka-topic script

kafka-topics --command-config=./kafka-debug.properties --list --bootstrap-server <External IP of Ingress>:443

The error that I am getting is

kafka-topics --command-config=./kafka-debug.properties --list --bootstrap-server 34.150.209.163:443
[2022-10-17 08:22:33,738] ERROR [AdminClient clientId=adminclient-1] Connection to node -1 (/34.150.209.163:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2022-10-17 08:22:33,739] WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205)
	at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:435)
	at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:523)
	at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:373)
	at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:293)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1408)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1339)
	at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:388)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:271)
	at java.base/sun.security.validator.Validator.validate(Validator.java:256)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329)
	... 19 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:383)
	... 24 more
Error while executing topic command : SSL handshake failed
[2022-10-17 08:22:33,742] ERROR org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205)
	at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:435)
	at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:523)
	at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:373)
	at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:293)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1408)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1339)
	at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:388)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:271)
	at java.base/sun.security.validator.Validator.validate(Validator.java:256)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329)
	... 19 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:383)
	... 24 more
 (kafka.admin.TopicCommand$)

rajjaiswalsaumya Oct 17, 2022
Author

Here is my created cluster services screenshot

rajjaiswalsaumya Oct 17, 2022
Author

Here is my ingress details.

scholzj Oct 17, 2022
Maintainer

The screenshots don't mean much as they are form some custom UI and it is not clear where does it take the columns and what do they mean. If you want to share how the resources look like, the best way is to use kubectl get ... -o yaml which produces the YAMl of the resource and makes it clear what it means.

Based on the YAML of the Kafka CR, you seem to be using very old version of Strimzi. That makes it hard to help because nobody remembers what was added since such an old version or what bugs there might have been. So ideally, you should upgrade to an latest version. In any case, your YAML has a lot of made up fields which do not exist in Strimzi. And I also wonder if you have the host field correctly => the host field should be an address under which your broker will be available to the clients through the Ingress. Normally, it would match some domain to which you can route the DNS. This is also what you have to use when you connect from the client. I have no idea where you got the IP address 34.150.209.163, but you need to connect to --bootstrap-server kafka-ccdf-2-kafka-bootstrap.sedjetstream-ccdf-dev-kafka-ccdf-2.svc:443 (and to be able to connect there, you of course need to make sure you have the DNS records for this domain name etc.).

rajjaiswalsaumya Oct 21, 2022
Author

I updated those url still this didnot work. Our operator team evaluated and confirmed strimzi + ingress with nginx on gcp does not work correctly. I will request if you can provide a sample working chart where you evaluated this on gcp will be helpful. We are using strimzi 0.27.1

kinihun · 2022-10-21T13:04:34Z

kinihun
Oct 21, 2022

You are missing the following configuration
class: nginx
brokerCertChainAndKey:
secretName: <>
certificate: tls.crt
key: tls.key

0 replies

rajjaiswalsaumya · 2022-10-22T00:18:21Z

rajjaiswalsaumya
Oct 22, 2022
Author

What does this secret name of ?

0 replies

Strimzi

Kafka external load balancer support on port 443 #7466

Uh oh!

rajjaiswalsaumya Oct 12, 2022

Replies: 3 comments · 12 replies

Uh oh!

scholzj Oct 12, 2022 Maintainer

Uh oh!

Uh oh!

rajjaiswalsaumya Oct 17, 2022 Author

Uh oh!

rajjaiswalsaumya Oct 17, 2022 Author

Uh oh!

rajjaiswalsaumya Oct 17, 2022 Author

Uh oh!

scholzj Oct 17, 2022 Maintainer

Uh oh!

Uh oh!

rajjaiswalsaumya Oct 21, 2022 Author

Uh oh!

kinihun Oct 21, 2022

Uh oh!

rajjaiswalsaumya Oct 22, 2022 Author

rajjaiswalsaumya
Oct 12, 2022

Replies: 3 comments 12 replies

scholzj
Oct 12, 2022
Maintainer

rajjaiswalsaumya Oct 17, 2022
Author

rajjaiswalsaumya Oct 17, 2022
Author

rajjaiswalsaumya Oct 17, 2022
Author

scholzj Oct 17, 2022
Maintainer

rajjaiswalsaumya Oct 21, 2022
Author

kinihun
Oct 21, 2022

rajjaiswalsaumya
Oct 22, 2022
Author