Principal = User:ANONYMOUS is Denied operation = DESCRIBE from host = 10.42.1.76 on resource

[ChiragR7]

Hi,

I have kafka configuration as below

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.8.0
    metadataVersion: 3.8-IV0
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        configuration:
          brokerCertChainAndKey:
            secretName: kafka-tls
            certificate: tls.crt
            key: tls.key
    config:
      default.replication.factor: 3
      min.insync.replicas: 2
      auto.create.topics.enable: "false"  
  entityOperator:
    topicOperator: {}
    userOperator: {}

now I am trying to enable authorization by adding

    authorization:
      type: simple

I have tried to define KafkaUser also

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: kafka-admin
  namespace: kafka
  labels:
    strimzi.io/cluster: kafka-cluster
spec:
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: "*"
          patternType: literal
        operations:
        - All
      - resource:
          type: group
          name: "*"
          patternType: literal
        operations:
        - All
      - resource:
          type: cluster
        operations:
        - All

On client side I do set

security.protocol=ssl
enable.ssl.certificate.verification=false

But still getting this error. Any help?
Note:- everything works until I add

 authorization:
      type: simple

[im-konge]

What error you are getting?

[ChiragR7]

I am trying to follow

https://strimzi.io/blog/2021/05/07/deploying-kafka-with-lets-encrypt-certificates/

it works on client side

But when I enable authorization I am getting error

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: kafka-admin
  namespace: kafka
  labels:
    strimzi.io/cluster: kafka-cluster
spec:
  authentication:
    type: tls-external
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: "*"
          patternType: prefix
        operations:
        - All
        host: "*"
      - resource:
          type: group
          name: "*"
          patternType: prefix
        operations:
        - All
        host: "*"

Certificate cerated using

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kafka-admin
  namespace: kafka
spec:
  secretName: kafka-admin
  privateKey:
    algorithm: ECDSA
    size: 384
  commonName: kafka-admin
  duration: 87600h # 3650 days (10 years)
  renewBefore: 720h # 30 days
  isCA: false
  usages:
    - server auth
    - client auth
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer
    group: cert-manager.io

I am getting

 Failed authentication with /10.42.0.0 (channelId=10.42.1.109:9094-10.42.0.0:35893-4) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [d │
│ ata-plane-kafka-network-thread-1-ListenerName(EXTERNAL-9094)-SSL-10]

Do I need to sign my certificate using sign it with strimzi-client-ca?

[im-konge]

That would suggest that you are using some wrong certificates or the certificates don't contain the name of the user itself.

[ChiragR7]

kubectl get secret kafka-admin -n kafka -o jsonpath='{.data.tls.crt}' | base64 --decode |openssl x509 -noout -subject
subject=CN = kafka-admin

[im-konge]

There can be more things actually:

• are you connecting to right port with that?
• you should use the cluster certificate and also the user certificate IIRC
• is there anything more in the Kafka logs?

Otherwise if you are using all the certificates correctly, it should be fine. As you can see, it's not about the authorization but authentication now. That points to some issue with how the certificates are used.

[im-konge]

Also, you should check that you are using correct security protocol etc.
Could you try to connect there using the openssl command as mentioned in the blogpost you posted above?

[adrian-salazar-klarrio]

Hello, I am having this same issue so I am reusing this discussion. This is my config:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: testcluster
  namespace: strimzi
spec:
  clientsCa:
    generateCertificateAuthority: false
  kafka:
    authorization:
      type: simple
      superUsers:
      - CN=super-user
    version: 3.7.1
    replicas: 3
    listeners:
    - name: tls
      port: 9092
      type: internal
      tls: true
      configuration:
        brokerCertChainAndKey:
          secretName: strimzi-server-certificate
          certificate: tls.crt
          key: tls.key
    config:
      ***omitted***
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
        class: kafka-v4
      - id: 1
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
        class: kafka-v4
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
      class: standard-retain-v1
  cruiseControl: {}
  entityOperator:
    userOperator: {}

The super user mentioned on top is referenced as

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: super-user
  namespace: strimzi
  labels:
    strimzi.io/cluster: testcluster
spec:
  authentication:
    type: tls-external

I have a secret in that ns called testcluster-clients-ca-cert with the ca.crt of my client secret and another secret called testcluster-clients-ca with some gibberish so the cluster operator doesn't complain.

Aside from that, I have a secret called superuser with the client certificate and a job that executes a bash script to create some ACL's as follows:

apiVersion: batch/v1
kind: Job
metadata:
  name: strimzi-acls-3.7.1
  namespace: strimzi
spec:
  template:
    spec:
      containers:
      - name: strimzi-acls
        command:
        - bash
        - '-e'
        - '-c'
        - |
          echo "ssl.keystore.location=/var/run/secrets/java.io/keystores/client/keystore.jks" > /tmp/kafka.client.config
          echo "ssl.truststore.location=/var/run/secrets/java.io/keystores/client/truststore.jks" >> /tmp/kafka.client.config
          echo "ssl.keystore.password=$JKS_PASSWORD" >> /tmp/kafka.client.config
          echo "ssl.truststore.password=$JKS_PASSWORD" >> /tmp/kafka.client.config
          echo "security.protocol=SSL" >> /tmp/kafka.client.config
          bash /home/acls.sh
          exit 0
        env:
        - name: BOOTSTRAP_SERVER
          value: testcluster-kafka-bootstrap.strimzi.svc:9092
          valueFrom:
            secretKeyRef:
              name: superuser
              key: password
              optional: false
        volumeMounts:
        - mountPath: /var/run/secrets/java.io/keystores/client
          name: client-ks-files
        - mountPath: /home
          name: strimzi-acls
      volumes:
      - name: client-ks-files
        secret:
          defaultMode: 420
          secretName: superuser
      - name: strimzi-acls
        configMap:
          name: strimzi-acls

A job that is almost same as this was used to create some topics before enabling authorization in the Kafka resource with success. The moment I added authorization -alongside my custom client CA and the KafkaUser- but I always get the same message as the title of this discussion.

The CN of my client cert is super-user as seen in

kubectl get secret superuser -o jsonpath='{.data.tls\.crt}' | base64 --decode |openssl x509 -noout -subject
subject=CN=super-user

I would expect a full Principal gere and not just User:ANONYMOUS after setting my client CA in the expected strimzi secret. Probably I am doing something wrong but I can't find the issue. Could you help here? Strimzi version used is 0.44.0

[im-konge]

I would firstly try to run the client outside of the Job and write what needs to be done in. order to connect with the super-user and after that I would create some Job that would do it.

[adrian-salazar-klarrio]

The tls config for my user is actually seen in the bash script, that's why I shared the job. The acls.sh script uses the kafka-acls.sh tool. The rest of the config is the default.

          echo "ssl.keystore.location=/var/run/secrets/java.io/keystores/client/keystore.jks" > /tmp/kafka.client.config
          echo "ssl.truststore.location=/var/run/secrets/java.io/keystores/client/truststore.jks" >> /tmp/kafka.client.config
          echo "ssl.keystore.password=$JKS_PASSWORD" >> /tmp/kafka.client.config
          echo "ssl.truststore.password=$JKS_PASSWORD" >> /tmp/kafka.client.config
          echo "security.protocol=SSL" >> /tmp/kafka.client.config
          bash /home/acls.sh

This config worked in the past using another kafka operator. But anyway, I'll manually configure a client and come back to you. Thanks for the prompt answer already.

[scholzj]

The main issue is that you don't have authentication enabled in the broker. You have to add it to the listener in the Kafka CR. Only then you can start looking at the client configuration.

[adrian-salazar-klarrio]

@scholzj Thanks!!!! I had another review to the docs and I saw I was missing in my listener:

      authentication:
        type: tls

This is not my answer -tho I thought it could be related- so I cannot mark it as answered. Thanks for the help to all.

[im-konge]

Good catch Jakub :) thanks