Unable to Connect kafka broker with TLS authetication Enabled

[shreyasarani23]

I have configured my kafka cluster with TLS authetication:

 listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls

And following is the configuration of my KafkaUser:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: my-kafka-user
  labels:
    strimzi.io/cluster: my-kafka-cluster
spec:
  authentication:
    type: tls

And I am using the built-in truststore which comes in with a JDK (Java) installation.
Following are the steps I followed for client credentials

1. export KAFKA_USER_NAME=my-kafka-user
2. kubectl get secret $KAFKA_USER_NAME -o jsonpath='{.data.user.crt}' -n kafka | base64 --decode > user.crt
3. kubectl get secret $KAFKA_USER_NAME -o jsonpath='{.data.user.key}' -n kafka | base64 --decode > user.key
4. kubectl get secret $KAFKA_USER_NAME -o jsonpath='{.data.user.p12}' -n kafka | base64 --decode > user.p12
5. kubectl get secret $KAFKA_USER_NAME -o jsonpath='{.data.user.password}' -n kafka | base64 --decode > user.password
6. export USER_P12_FILE_PATH=user.p12
7. export USER_KEY_PASSWORD_FILE_PATH=user.password
8. export KEYSTORE_NAME=kafka-keystore.jks
9. export KEYSTORE_PASSWORD=test123
10. export PASSWORD=cat $USER_KEY_PASSWORD_FILE_PATH
11. sudo keytool -importkeystore -deststorepass $KEYSTORE_PASSWORD -destkeystore $KEYSTORE_NAME -srckeystore $USER_P12_FILE_PATH -srcstorepass $PASSWORD -srcstoretype PKCS12
12. sudo keytool -list -alias $KAFKA_USER_NAME -keystore $KEYSTORE_NAME

Following are the steps I followed for server credentials
1.export CLUSTER_NAME=my-kafka-cluster
2.kubectl get secret $CLUSTER_NAME-cluster-ca-cert -o jsonpath='{.data.ca.crt}' -n kafka | base64 --decode > ca.crt
3. kubectl get secret $CLUSTER_NAME-cluster-ca-cert -o jsonpath='{.data.ca.password}' -n kafka | base64 --decode > ca.password
4. export CERT_FILE_PATH=ca.crt
5. export CERT_PASSWORD_FILE_PATH=ca.password
6. export KEYSTORE_LOCATION=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts
7. sudo keytool -importcert -alias server-certificate -file $CERT_FILE_PATH -keystore $KEYSTORE_LOCATION -keypass $PASSWORD
8. sudo keytool -list -alias server-certificate -keystore $KEYSTORE_LOCATION

Following is the configuration of my properties file

bootstrap.servers=10.0.147.119:9093 //cluster IP of bootstrap server followed by port number
security.protocol=SSL
ssl.truststore.location=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts
ssl.truststore.password=changeit
ssl.keystore.location=kafka-keystore.jks
ssl.keystore.password=test123
ssl.key.password=INgCltezG7tL  // content of user.password file

I have installed kafka on my local machine and running the consumer with the below commands:

1.export KAFKA_HOME=/home/shreyas/Documents/Kafka/kafka_2.13-2.7.0
2. export TOPIC_NAME=new-topic

$KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap-server 10.0.147.119:9093 --topic $TOPIC_NAME --consumer.config client-ssl-auth.properties --from-beginning

And I am getting the following error.

[2021-05-21 11:09:16,311] WARN [Consumer clientId=consumer-console-consumer-66980-1, groupId=console-consumer-66980] Bootstrap broker 10.0.147.119:9093 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-05-21 11:09:35,883] WARN [Consumer clientId=consumer-console-consumer-66980-1, groupId=console-consumer-66980] Bootstrap broker 10.0.147.119:9093 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-05-21 11:10:11,074] WARN [Consumer clientId=consumer-console-consumer-66980-1, groupId=console-consumer-66980] Bootstrap broker 10.0.147.119:9093 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-05-21 11:11:41,676] WARN [Consumer clientId=consumer-console-consumer-66980-1, groupId=console-consumer-66980] Bootstrap broker 10.0.147.119:9093 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2021-05-21 11:13:52,805] WARN [Consumer clientId=consumer-console-consumer-66980-1, groupId=console-consumer-66980] Connection to node -1 (/10.0.147.119:9093) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2021-05-21 11:13:52,806] WARN [Consumer clientId=consumer-console-consumer-66980-1, groupId=console-consumer-66980] Bootstrap broker 10.0.147.119:9093 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

I don't know why I am getting this error. For your information the kafka brokers are up and running. Please let me know if I am missing something or some configuration is incorrect.

[scholzj]

If you want to access Kafka from outside, you will need to configure an external listener - there are different types, you have to choose the one which works for your environment. All the types and options are described in the docs, there was also a 5 part blog post series on the Strimzi blog about external listeners. So that should get you started.

[shreyasarani23]

Thanks @scholzj I know the fact assigning external listerner with public IP, nodeport etc works. But I have a constrain that I will not be exposing my kafka cluster outside the kubernetes. Kafka cluster should be accessible within the kubernetes cluster. And I know that I am trying to accessing kafka bootstrap with it's clusterip which in this case will not work because I am externally trying to access it. My requirement is that without using public IP I want to achieve TLS authetication internally So is there any other way?

[scholzj]

Well, you said this:

I have installed kafka on my local machine and running the consumer with the below commands:

So that suggests that you are trying to access a Kafka cluster in Kubernetes from your local machine. That needs external listener. The external listener does not depend on any public IPs - it is for exposing the Kafka cluster outside of Kubernetes even if that would be with private IPs visible only to your network etc. If you do not want to use it, you have to run everything from some pod in your Kube cluster and not from your local machine.

[shreyasarani23]

Oh u mean to say that some other kubernetes pod should be running a kafka image which in this case will be created as client. And from this client pod I should carry out all the operations like creating keystore for client, importing certificate to truststore etc and run the consumer on the same client pod?

[scholzj]

You can run whatever image you want in the pod with whatever software you want - there is no limitation to it. But the internal listeners in Kafka are accessible only from other Kubernetes pods. If you want to access it from outside of Kubernetes, you need to use the external listener. This is caused by Kafka's internal discovery mechanisms.

[shreyasarani23]

Ok got it @scholzj