After Kraft Migration "java.lang.IllegalArgumentException: Invalid connection string." for tieredstorage

[P2W2]

Hello,
we migrated Kafka from a zookeeper deployment to a Kraft.
We using the tieredStorage option with Azure.
Now we getting a "java.lang.IllegalArgumentException: Invalid connection string." Error.
If I add the Connection string directly it is working without a Secret the configuration is working.
I think my env. variable is in the wrong format but I can't figure out which is the right one for a configuration with a nodepool.
This Kafka Configuration was working for Zookeeper:

---
# Source: strimzi-kafka-operator/templates/kafka-cluster.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name:-kafka-cluster
  annotations:
    strimzi.io/kraft: "enabled"
    strimzi.io/node-pools: "enabled"
spec:
  kafka:
    version: 3.9.0
    listeners: 
      - name: plain
        port: 9092
        tls: false
        type: internal
      - authentication:
          type: scram-sha-512
        name: tls
        port: 9093
        tls: true
        type: internal
    authorization:
      type: simple
    tieredStorage:
      type: custom
      remoteStorageManager:
        className: io.aiven.kafka.tieredstorage.RemoteStorageManager
        classPath: /opt/kafka/plugins/tiered-storage/*
        config: 
          chunk.size: "5242880"
          storage.azure.connection.string: ${env:AZURE_CONNECTION_STRING}
          storage.azure.container.name:-devddc-kafka-strorage
          storage.backend.class: io.aiven.kafka.tieredstorage.storage.azure.AzureBlobStorage
    config: 
      auto.create.topic.enable: "false"
      config.providers: env
      config.providers.env.class: io.strimzi.kafka.EnvVarConfigProvider
      controller.quorum.election.timeout.ms: 3000
      controller.quorum.request.timeout.ms: 3000
      controller.quorum.retry.backoff.ms: 200
      default.replication.factor: 3
      fetch.message.max.bytes: 5000000
      log.retention.ms: 43200000
      log.roll.jitter.ms: 60000
      log.roll.ms: 300000
      log.segment.bytes: 134217728
      max.request.size: 5000000
      message.max.bytes: 5000000
      min.insync.replicas: 2
      offsets.topic.replication.factor: 3
      producer.max.request.size: 5000000
      remote.log.storage.manager.class.name: io.aiven.kafka.tieredstorage.RemoteStorageManager
      remote.log.storage.manager.class.path: /opt/kafka/plugins/tiered-storage/*
      remote.log.storage.system.enable: "true"
      remote.storage.enable: "true"
      replica.fetch.max.bytes: 5000000
      rlmm.config.remote.log.metadata.topic.replication.factor: 1
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    template:
      pod:
        imagePullSecrets: 
          - name:-acr-images-creds-secret
      kafkaContainer:
        env: 
          - name: AZURE_CONNECTION_STRING
            valueFrom:
              secretKeyRef:
                key: DEV_KAFKA_STORAGE_CONNECTION_STRING
                name: kafka-strimzi-kafka-operator-vault-secrets
    image: crmegahubwesteurope.azurecr.io/-devddc/images/-kafka:3.9.0
    metricsConfig: ...
  entityOperator: ...
    userOperator: ...
---
# Source: strimzi-kafka-operator/templates/kafka-node-pool.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controller
  labels:
    strimzi.io/cluster: kafka-cluster
spec:
  replicas: 3
  resources:
    requests:
      memory: "3Gi"
      cpu: "700m"
    limits:
      memory: "4Gi"
      cpu: "1000m"
  roles:
    - controller
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: "20Gi"
        kraftMetadata: shared
        deleteClaim: false
  template:
    pod:
      metadata:
        annotations:
          prometheus.io/path: /metrics
          prometheus.io/port: "9404"
          prometheus.io/scrape: "true"
    kafkaContainer:
      env:
        - name: AZURE_CONNECTION_STRING
          valueFrom:
            secretKeyRef:
              name: "kafka-strimzi-kafka-operator-vault-secrets"
              key: "DEV_KAFKA_STORAGE_CONNECTION_STRING"

[scholzj]

I guess a good start would be to share the full logs to understand where does the error happen and where does it come from.

[scholzj]

Also, you do not seem to have any brokers. And a cluster name starting with - seems a bit strange. Is that really allowed?

[P2W2]

Sorry, this is not the full cluster name I just removed internal names that I don't want to post here, so it is not starting with "-".
Full Error Log:

removed directory '/tmp/hsperfdata_kafka'
removed '/tmp/kafka/strimzi.kafka.metadata.config.state'
removed '/tmp/kafka/clients.truststore.p12'
removed '/tmp/kafka/cluster.keystore.p12'
removed '/tmp/kafka/cluster.truststore.p12'
removed directory '/tmp/kafka'
removed '/tmp/kafka-agent.properties'
removed '/tmp/strimzi.properties'
STRIMZI_BROKER_ID=0
Preparing truststore for replication listener
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for replication listener is complete
Looking for the CA matching the server certificate
CA matching the server certificate found: /opt/kafka/cluster-ca-certs/ca.crt
Preparing keystore for replication and clienttls listener
Preparing keystore for replication and clienttls listener is complete
Preparing truststore for client authentication
Adding /opt/kafka/client-ca-certs/ca.crt to truststore /tmp/kafka/clients.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for client authentication is complete
Starting Kafka with configuration:
##############################
##############################
# This file is automatically generated by the Strimzi Cluster Operator
# Any changes to this file will be ignored and overwritten!
##############################
##############################

##########
# Node / Broker ID
##########
node.id=0

##########
# Kafka message logs configuration
##########
log.dirs=/var/lib/kafka/data-0/kafka-log0

##########
# Control Plane listener
##########
...

##########
# Replication listener
##########
...

##########
# Listener configuration: PLAIN-9092
##########

##########
# Listener configuration: TLS-9093
##########
...


##########
# Common listener configuration
##########
...

##########
# Authorization
##########
...

##########
# Kafka tiered storage configuration
##########
# RLMM configuration generated by Strimzi
remote.log.storage.system.enable=true
remote.log.metadata.manager.impl.prefix=rlmm.config.
remote.log.metadata.manager.class.name=org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
remote.log.metadata.manager.listener.name=REPLICATION-9091
rlmm.config.remote.log.metadata.common.client.bootstrap.servers=-kafka-cluster-kafka-brokers:9091
rlmm.config.remote.log.metadata.common.client.security.protocol=SSL
rlmm.config.remote.log.metadata.common.client.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
rlmm.config.remote.log.metadata.common.client.ssl.keystore.password=[hidden]
rlmm.config.remote.log.metadata.common.client.ssl.keystore.type=PKCS12
rlmm.config.remote.log.metadata.common.client.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
rlmm.config.remote.log.metadata.common.client.ssl.truststore.password=[hidden]
rlmm.config.remote.log.metadata.common.client.ssl.truststore.type=PKCS12
# RSM configs set by the operator and by the user
remote.log.storage.manager.class.name=io.aiven.kafka.tieredstorage.RemoteStorageManager
remote.log.storage.manager.class.path=/opt/kafka/plugins/tiered-storage/*
remote.log.storage.manager.impl.prefix=rsm.config.
rsm.config.chunk.size=5242880
rsm.config.storage.azure.connection.string=${env:AZURE_CONNECTION_STRING}
rsm.config.storage.azure.container.name=-devddc-kafka-strorage
rsm.config.storage.backend.class=io.aiven.kafka.tieredstorage.storage.azure.AzureBlobStorage

##########
# Config providers
##########
# Configuration providers configured by the user and by Strimzi
config.providers=env,strimzienv,strimzifile,strimzidir
config.providers.strimzienv.class=org.apache.kafka.common.config.provider.EnvVarConfigProvider
config.providers.strimzienv.param.allowlist.pattern=.*
config.providers.strimzifile.class=org.apache.kafka.common.config.provider.FileConfigProvider
config.providers.strimzifile.param.allowed.paths=/opt/kafka
config.providers.strimzidir.class=org.apache.kafka.common.config.provider.DirectoryConfigProvider
config.providers.strimzidir.param.allowed.paths=/opt/kafka

##########
# User provided configuration
##########
auto.create.topic.enable=false
config.providers.env.class=io.strimzi.kafka.EnvVarConfigProvider
controller.quorum.election.timeout.ms=3000
default.replication.factor=3
fetch.message.max.bytes=5000000
log.retention.ms=43200000
log.roll.jitter.ms=60000
log.roll.ms=300000
log.segment.bytes=134217728
max.request.size=5000000
message.max.bytes=5000000
min.insync.replicas=2
offsets.topic.replication.factor=3
producer.max.request.size=5000000
remote.log.storage.manager.class.name=io.aiven.kafka.tieredstorage.RemoteStorageManager
remote.log.storage.manager.class.path=/opt/kafka/plugins/tiered-storage/*
remote.log.storage.system.enable=true
remote.storage.enable=true
replica.fetch.max.bytes=5000000
rlmm.config.remote.log.metadata.topic.replication.factor=1
transaction.state.log.min.isr=2
transaction.state.log.replication.factor=3


##########
# KRaft configuration
##########
process.roles=broker
controller.listener.names=CONTROLPLANE-9090
controller.quorum.voters=4@-kafka-cluster-controller-4.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local:9090,5@-kafka-cluster-controller-5.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local:9090,6@-kafka-cluster-controller-6.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local:9090

##########
# KRaft metadata log dir configuration
##########
metadata.log.dir=/var/lib/kafka/data-0/kafka-log0
Configuring Java heap: -Xms3113851289 -Xmx3113851289
Kafka metadata config state [4]
Using KRaft [true]
Making sure the Kraft storage is formatted with cluster ID 7wDfh2TwQPayDWMrH1i0UQ and metadata version 3.9
2025-06-17 07:43:27,792 INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$) [main]
2025-06-17 07:43:28,066 INFO Configuring EnvVar config provider: {} (io.strimzi.kafka.EnvVarConfigProvider) [main]
2025-06-17 07:43:28,072 INFO Closing EnvVar config provider (io.strimzi.kafka.EnvVarConfigProvider) [main]
2025-06-17 07:43:28,079 INFO KafkaConfig values:
	advertised.listeners = REPLICATION-9091://-kafka-cluster-broker-0.-kafka-cluster-kafka-brokers.-kafka.svc:9091,PLAIN-9092://-kafka-cluster-broker-0.-kafka-cluster-kafka-brokers.-kafka.svc:9092,TLS-9093://-kafka-cluster-broker-0.-kafka-cluster-kafka-brokers.-kafka.svc:9093
	alter.config.policy.class.name = null
	alter.log.dirs.replication.quota.window.num = 11
	alter.log.dirs.replication.quota.window.size.seconds = 1
	authorizer.class.name = org.apache.kafka.metadata.authorizer.StandardAuthorizer
	auto.create.topics.enable = true
	auto.include.jmx.reporter = true
	auto.leader.rebalance.enable = true
	background.threads = 10
	broker.heartbeat.interval.ms = 2000
	broker.id = 0
	broker.id.generation.enable = true
	broker.rack = null
	broker.session.timeout.ms = 9000
	client.quota.callback.class = null
	compression.gzip.level = -1
	compression.lz4.level = 9
	compression.type = producer
	compression.zstd.level = 3
	connection.failed.authentication.delay.ms = 100
	connections.max.idle.ms = 600000
	connections.max.reauth.ms = 0
	control.plane.listener.name = null
	controlled.shutdown.enable = true
	controlled.shutdown.max.retries = 3
	controlled.shutdown.retry.backoff.ms = 5000
	controller.listener.names = CONTROLPLANE-9090
	controller.quorum.append.linger.ms = 25
	controller.quorum.bootstrap.servers = []
	controller.quorum.election.backoff.max.ms = 1000
	controller.quorum.election.timeout.ms = 3000
	controller.quorum.fetch.timeout.ms = 2000
	controller.quorum.request.timeout.ms = 2000
	controller.quorum.retry.backoff.ms = 20
	controller.quorum.voters = [4@-kafka-cluster-controller-4.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local:9090, 5@-kafka-cluster-controller-5.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local:9090, 6@-kafka-cluster-controller-6.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local:9090]
	controller.quota.window.num = 11
	controller.quota.window.size.seconds = 1
	controller.socket.timeout.ms = 30000
	create.topic.policy.class.name = null
	default.replication.factor = 3
	delegation.token.expiry.check.interval.ms = 3600000
	delegation.token.expiry.time.ms = 86400000
	delegation.token.master.key = null
	delegation.token.max.lifetime.ms = 604800000
	delegation.token.secret.key = null
	delete.records.purgatory.purge.interval.requests = 1
	delete.topic.enable = true
	early.start.listeners = null
	eligible.leader.replicas.enable = false
	fetch.max.bytes = 57671680
	fetch.purgatory.purge.interval.requests = 1000
	group.consumer.assignors = [org.apache.kafka.coordinator.group.assignor.UniformAssignor, org.apache.kafka.coordinator.group.assignor.RangeAssignor]
	group.consumer.heartbeat.interval.ms = 5000
	group.consumer.max.heartbeat.interval.ms = 15000
	group.consumer.max.session.timeout.ms = 60000
	group.consumer.max.size = 2147483647
	group.consumer.migration.policy = disabled
	group.consumer.min.heartbeat.interval.ms = 5000
	group.consumer.min.session.timeout.ms = 45000
	group.consumer.session.timeout.ms = 45000
	group.coordinator.append.linger.ms = 10
	group.coordinator.new.enable = false
	group.coordinator.rebalance.protocols = [classic]
	group.coordinator.threads = 1
	group.initial.rebalance.delay.ms = 3000
	group.max.session.timeout.ms = 1800000
	group.max.size = 2147483647
	group.min.session.timeout.ms = 6000
	group.share.delivery.count.limit = 5
	group.share.enable = false
	group.share.heartbeat.interval.ms = 5000
	group.share.max.groups = 10
	group.share.max.heartbeat.interval.ms = 15000
	group.share.max.record.lock.duration.ms = 60000
	group.share.max.session.timeout.ms = 60000
	group.share.max.size = 200
	group.share.min.heartbeat.interval.ms = 5000
	group.share.min.record.lock.duration.ms = 15000
	group.share.min.session.timeout.ms = 45000
	group.share.partition.max.record.locks = 200
	group.share.record.lock.duration.ms = 30000
	group.share.session.timeout.ms = 45000
	initial.broker.registration.timeout.ms = 60000
	inter.broker.listener.name = REPLICATION-9091
	inter.broker.protocol.version = 3.9-IV0
	kafka.metrics.polling.interval.secs = 10
	kafka.metrics.reporters = []
	leader.imbalance.check.interval.seconds = 300
	leader.imbalance.per.broker.percentage = 10
	listener.security.protocol.map = CONTROLPLANE-9090:SSL,REPLICATION-9091:SSL,PLAIN-9092:PLAINTEXT,TLS-9093:SASL_SSL
	listeners = REPLICATION-9091://0.0.0.0:9091,PLAIN-9092://0.0.0.0:9092,TLS-9093://0.0.0.0:9093
	log.cleaner.backoff.ms = 15000
	log.cleaner.dedupe.buffer.size = 134217728
	log.cleaner.delete.retention.ms = 86400000
	log.cleaner.enable = true
	log.cleaner.io.buffer.load.factor = 0.9
	log.cleaner.io.buffer.size = 524288
	log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
	log.cleaner.max.compaction.lag.ms = 9223372036854775807
	log.cleaner.min.cleanable.ratio = 0.5
	log.cleaner.min.compaction.lag.ms = 0
	log.cleaner.threads = 1
	log.cleanup.policy = [delete]
	log.dir = /tmp/kafka-logs
	log.dir.failure.timeout.ms = 30000
	log.dirs = /var/lib/kafka/data-0/kafka-log0
	log.flush.interval.messages = 9223372036854775807
	log.flush.interval.ms = null
	log.flush.offset.checkpoint.interval.ms = 60000
	log.flush.scheduler.interval.ms = 9223372036854775807
	log.flush.start.offset.checkpoint.interval.ms = 60000
	log.index.interval.bytes = 4096
	log.index.size.max.bytes = 10485760
	log.initial.task.delay.ms = 30000
	log.local.retention.bytes = -2
	log.local.retention.ms = -2
	log.message.downconversion.enable = true
	log.message.format.version = 3.0-IV1
	log.message.timestamp.after.max.ms = 9223372036854775807
	log.message.timestamp.before.max.ms = 9223372036854775807
	log.message.timestamp.difference.max.ms = 9223372036854775807
	log.message.timestamp.type = CreateTime
	log.preallocate = false
	log.retention.bytes = -1
	log.retention.check.interval.ms = 300000
	log.retention.hours = 168
	log.retention.minutes = null
	log.retention.ms = 43200000
	log.roll.hours = 168
	log.roll.jitter.hours = 0
	log.roll.jitter.ms = 60000
	log.roll.ms = 300000
	log.segment.bytes = 134217728
	log.segment.delete.delay.ms = 60000
	max.connection.creation.rate = 2147483647
	max.connections = 2147483647
	max.connections.per.ip = 2147483647
	max.connections.per.ip.overrides =
	max.incremental.fetch.session.cache.slots = 1000
	max.request.partition.size.limit = 2000
	message.max.bytes = 5000000
	metadata.log.dir = /var/lib/kafka/data-0/kafka-log0
	metadata.log.max.record.bytes.between.snapshots = 20971520
	metadata.log.max.snapshot.interval.ms = 3600000
	metadata.log.segment.bytes = 1073741824
	metadata.log.segment.min.bytes = 8388608
	metadata.log.segment.ms = 604800000
	metadata.max.idle.interval.ms = 500
	metadata.max.retention.bytes = 104857600
	metadata.max.retention.ms = 604800000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	min.insync.replicas = 2
	node.id = 0
	num.io.threads = 8
	num.network.threads = 3
	num.partitions = 1
	num.recovery.threads.per.data.dir = 1
	num.replica.alter.log.dirs.threads = null
	num.replica.fetchers = 1
	offset.metadata.max.bytes = 4096
	offsets.commit.required.acks = -1
	offsets.commit.timeout.ms = 5000
	offsets.load.buffer.size = 5242880
	offsets.retention.check.interval.ms = 600000
	offsets.retention.minutes = 10080
	offsets.topic.compression.codec = 0
	offsets.topic.num.partitions = 50
	offsets.topic.replication.factor = 3
	offsets.topic.segment.bytes = 104857600
	password.encoder.cipher.algorithm = AES/CBC/PKCS5Padding
	password.encoder.iterations = 4096
	password.encoder.key.length = 128
	password.encoder.keyfactory.algorithm = null
	password.encoder.old.secret = null
	password.encoder.secret = null
	principal.builder.class = class org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder
	process.roles = [broker]
	producer.id.expiration.check.interval.ms = 600000
	producer.id.expiration.ms = 86400000
	producer.purgatory.purge.interval.requests = 1000
	queued.max.request.bytes = -1
	queued.max.requests = 500
	quota.window.num = 11
	quota.window.size.seconds = 1
	remote.fetch.max.wait.ms = 500
	remote.log.index.file.cache.total.size.bytes = 1073741824
	remote.log.manager.copier.thread.pool.size = -1
	remote.log.manager.copy.max.bytes.per.second = 9223372036854775807
	remote.log.manager.copy.quota.window.num = 11
	remote.log.manager.copy.quota.window.size.seconds = 1
	remote.log.manager.expiration.thread.pool.size = -1
	remote.log.manager.fetch.max.bytes.per.second = 9223372036854775807
	remote.log.manager.fetch.quota.window.num = 11
	remote.log.manager.fetch.quota.window.size.seconds = 1
	remote.log.manager.task.interval.ms = 30000
	remote.log.manager.task.retry.backoff.max.ms = 30000
	remote.log.manager.task.retry.backoff.ms = 500
	remote.log.manager.task.retry.jitter = 0.2
	remote.log.manager.thread.pool.size = 10
	remote.log.metadata.custom.metadata.max.bytes = 128
	remote.log.metadata.manager.class.name = org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
	remote.log.metadata.manager.class.path = null
	remote.log.metadata.manager.impl.prefix = rlmm.config.
	remote.log.metadata.manager.listener.name = REPLICATION-9091
	remote.log.reader.max.pending.tasks = 100
	remote.log.reader.threads = 10
	remote.log.storage.manager.class.name = io.aiven.kafka.tieredstorage.RemoteStorageManager
	remote.log.storage.manager.class.path = /opt/kafka/plugins/tiered-storage/*
	remote.log.storage.manager.impl.prefix = rsm.config.
	remote.log.storage.system.enable = true
	replica.fetch.backoff.ms = 1000
	replica.fetch.max.bytes = 5000000
	replica.fetch.min.bytes = 1
	replica.fetch.response.max.bytes = 10485760
	replica.fetch.wait.max.ms = 500
	replica.high.watermark.checkpoint.interval.ms = 5000
	replica.lag.time.max.ms = 30000
	replica.selector.class = null
	replica.socket.receive.buffer.bytes = 65536
	replica.socket.timeout.ms = 30000
	replication.quota.window.num = 11
	replication.quota.window.size.seconds = 1
	request.timeout.ms = 30000
	reserved.broker.max.id = 1000
	sasl.client.callback.handler.class = null
	sasl.enabled.mechanisms = []
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.principal.to.local.rules = [DEFAULT]
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.connect.timeout.ms = null
	sasl.login.read.timeout.ms = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.login.retry.backoff.max.ms = 10000
	sasl.login.retry.backoff.ms = 100
	sasl.mechanism.controller.protocol = GSSAPI
	sasl.mechanism.inter.broker.protocol = GSSAPI
	sasl.oauthbearer.clock.skew.seconds = 30
	sasl.oauthbearer.expected.audience = null
	sasl.oauthbearer.expected.issuer = null
	sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
	sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
	sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
	sasl.oauthbearer.jwks.endpoint.url = null
	sasl.oauthbearer.scope.claim.name = scope
	sasl.oauthbearer.sub.claim.name = sub
	sasl.oauthbearer.token.endpoint.url = null
	sasl.server.callback.handler.class = null
	sasl.server.max.receive.size = 524288
	security.inter.broker.protocol = PLAINTEXT
	security.providers = null
	server.max.startup.time.ms = 9223372036854775807
	socket.connection.setup.timeout.max.ms = 30000
	socket.connection.setup.timeout.ms = 10000
	socket.listen.backlog.size = 50
	socket.receive.buffer.bytes = 102400
	socket.request.max.bytes = 104857600
	socket.send.buffer.bytes = 102400
	ssl.allow.dn.changes = false
	ssl.allow.san.changes = false
	ssl.cipher.suites = []
	ssl.client.auth = none
	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
	ssl.endpoint.identification.algorithm = HTTPS
	ssl.engine.factory.class = null
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.certificate.chain = null
	ssl.keystore.key = null
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.principal.mapping.rules = DEFAULT
	ssl.protocol = TLSv1.3
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.certificates = null
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	telemetry.max.bytes = 1048576
	transaction.abort.timed.out.transaction.cleanup.interval.ms = 10000
	transaction.max.timeout.ms = 900000
	transaction.partition.verification.enable = true
	transaction.remove.expired.transaction.cleanup.interval.ms = 3600000
	transaction.state.log.load.buffer.size = 5242880
	transaction.state.log.min.isr = 2
	transaction.state.log.num.partitions = 50
	transaction.state.log.replication.factor = 3
	transaction.state.log.segment.bytes = 104857600
	transactional.id.expiration.ms = 604800000
	unclean.leader.election.enable = false
	unclean.leader.election.interval.ms = 300000
	unstable.api.versions.enable = false
	unstable.feature.versions.enable = false
	zookeeper.clientCnxnSocket = null
	zookeeper.connect = null
	zookeeper.connection.timeout.ms = null
	zookeeper.max.in.flight.requests = 10
	zookeeper.metadata.migration.enable = false
	zookeeper.metadata.migration.min.batch.size = 200
	zookeeper.session.timeout.ms = 18000
	zookeeper.set.acl = false
	zookeeper.ssl.cipher.suites = null
	zookeeper.ssl.client.enable = false
	zookeeper.ssl.crl.enable = false
	zookeeper.ssl.enabled.protocols = null
	zookeeper.ssl.endpoint.identification.algorithm = HTTPS
	zookeeper.ssl.keystore.location = null
	zookeeper.ssl.keystore.password = null
	zookeeper.ssl.keystore.type = null
	zookeeper.ssl.ocsp.enable = false
	zookeeper.ssl.protocol = TLSv1.2
	zookeeper.ssl.truststore.location = null
	zookeeper.ssl.truststore.password = null
	zookeeper.ssl.truststore.type = null
 (kafka.server.KafkaConfig) [main]
2025-06-17 07:43:28,116 INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [main]
All of the log directories are already formatted.
KRaft storage formatting is done
Removing quorum-state file

Preparing Kafka Agent configuration

+ exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/kafka-server-start.sh /tmp/strimzi.properties
2025-06-17 07:43:29,773 INFO Starting KafkaAgent with brokerReadyFile=null, sessionConnectedFile=null, sslKeyStorePath=/tmp/kafka/cluster.keystore.p12, sslTrustStore=/tmp/kafka/cluster.truststore.p12 (io.strimzi.kafka.agent.KafkaAgent) [main]
2025-06-17 07:43:29,780 INFO Logging initialized @845ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log) [main]
2025-06-17 07:43:29,886 INFO jetty-9.4.56.v20240826; built: 2024-08-26T17:15:05.868Z; git: ec6782ff5ead824dabdcf47fa98f90a4aedff401; jvm 17.0.14+7-LTS (org.eclipse.jetty.server.Server) [main]
2025-06-17 07:43:29,904 INFO Started o.e.j.s.h.ContextHandler@45385f75{/v1/broker-state,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler) [main]
2025-06-17 07:43:29,904 INFO Started o.e.j.s.h.ContextHandler@1c9b0314{/v1/ready,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler) [main]
2025-06-17 07:43:29,904 INFO Started o.e.j.s.h.ContextHandler@49c90a9c{/v1/kraft-migration,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler) [main]
2025-06-17 07:43:30,201 INFO x509=X509@1838ccb8(-kafka-cluster-broker-0,h=[-kafka-cluster-kafka-bootstrap.-kafka.svc.cluster.local, -kafka-cluster-kafka-brokers, -kafka-cluster-kafka-brokers.-kafka, -kafka-cluster-kafka-bootstrap.-kafka.svc, -kafka-cluster-broker-0.-kafka-cluster-kafka-brokers.-kafka.svc, -kafka-cluster-kafka-brokers.-kafka.svc, -kafka-cluster-kafka-brokers.-kafka.svc.cluster.local, -kafka-cluster-kafka-bootstrap.-kafka, -kafka-cluster-broker-0.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local, -kafka-cluster-kafka-bootstrap, -kafka-cluster-kafka],a=[],w=[]) for Server@6c2ed0cd[provider=null,keyStore=file:///tmp/kafka/cluster.keystore.p12,trustStore=file:///tmp/kafka/cluster.truststore.p12] (org.eclipse.jetty.util.ssl.SslContextFactory) [main]
2025-06-17 07:43:30,380 INFO Started ServerConnector@26abb146{SSL, (ssl, http/1.1)}{0.0.0.0:8443} (org.eclipse.jetty.server.AbstractConnector) [main]
2025-06-17 07:43:30,382 INFO Started ServerConnector@14dd7b39{HTTP/1.1, (http/1.1)}{localhost:8080} (org.eclipse.jetty.server.AbstractConnector) [main]
2025-06-17 07:43:30,382 INFO Started @1446ms (org.eclipse.jetty.server.Server) [main]
2025-06-17 07:43:30,382 INFO Starting metrics registry (io.strimzi.kafka.agent.KafkaAgent) [main]
2025-06-17 07:43:30,387 INFO Found class org.apache.kafka.server.metrics.KafkaYammerMetrics for Kafka 3.3 and newer. (io.strimzi.kafka.agent.KafkaAgent) [main]
2025-06-17 07:43:30,487 INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$) [main]
2025-06-17 07:43:30,789 INFO Configuring EnvVar config provider: {} (io.strimzi.kafka.EnvVarConfigProvider) [main]
2025-06-17 07:43:30,791 INFO Closing EnvVar config provider (io.strimzi.kafka.EnvVarConfigProvider) [main]
2025-06-17 07:43:30,889 INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [main]
2025-06-17 07:43:31,112 INFO Configuring EnvVar config provider: {} (io.strimzi.kafka.EnvVarConfigProvider) [main]
2025-06-17 07:43:31,113 INFO Closing EnvVar config provider (io.strimzi.kafka.EnvVarConfigProvider) [main]
2025-06-17 07:43:31,176 INFO Configuring EnvVar config provider: {} (io.strimzi.kafka.EnvVarConfigProvider) [main]
2025-06-17 07:43:31,177 INFO Closing EnvVar config provider (io.strimzi.kafka.EnvVarConfigProvider) [main]
2025-06-17 07:43:31,276 INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler) [main]
2025-06-17 07:43:31,280 INFO [BrokerServer id=0] Transition from SHUTDOWN to STARTING (kafka.server.BrokerServer) [main]
2025-06-17 07:43:31,281 INFO [SharedServer id=0] Starting SharedServer (kafka.server.SharedServer) [main]
2025-06-17 07:43:31,281 INFO Configuring EnvVar config provider: {} (io.strimzi.kafka.EnvVarConfigProvider) [main]
2025-06-17 07:43:31,282 INFO Closing EnvVar config provider (io.strimzi.kafka.EnvVarConfigProvider) [main]
2025-06-17 07:43:31,392 INFO [LogLoader partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log0] Recovering unflushed segment 7856. 0/1 recovered for __cluster_metadata-0. (kafka.log.LogLoader) [main]
2025-06-17 07:43:31,393 INFO [LogLoader partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 7856 with message format version 2 (kafka.log.UnifiedLog$) [main]
2025-06-17 07:43:31,394 INFO [LogLoader partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log0] Reloading from producer snapshot and rebuilding producer state from offset 7856 (kafka.log.UnifiedLog$) [main]
2025-06-17 07:43:31,394 INFO Deleted producer state snapshot /var/lib/kafka/data-0/kafka-log0/__cluster_metadata-0/00000000000000013660.snapshot (org.apache.kafka.storage.internals.log.SnapshotFile) [main]
2025-06-17 07:43:31,394 INFO Deleted producer state snapshot /var/lib/kafka/data-0/kafka-log0/__cluster_metadata-0/00000000000000014303.snapshot (org.apache.kafka.storage.internals.log.SnapshotFile) [main]
2025-06-17 07:43:31,412 INFO [ProducerStateManager partition=__cluster_metadata-0] Wrote producer snapshot at offset 7856 with 0 producer ids in 16 ms. (org.apache.kafka.storage.internals.log.ProducerStateManager) [main]
2025-06-17 07:43:31,412 INFO [LogLoader partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log0] Producer state recovery took 0ms for snapshot load and 18ms for segment recovery from offset 7856 (kafka.log.UnifiedLog$) [main]
2025-06-17 07:43:31,584 INFO [ProducerStateManager partition=__cluster_metadata-0] Wrote producer snapshot at offset 14303 with 0 producer ids in 15 ms. (org.apache.kafka.storage.internals.log.ProducerStateManager) [main]
2025-06-17 07:43:31,590 INFO [LogLoader partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log0] Loading producer state till offset 14303 with message format version 2 (kafka.log.UnifiedLog$) [main]
2025-06-17 07:43:31,590 INFO [LogLoader partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log0] Reloading from producer snapshot and rebuilding producer state from offset 14303 (kafka.log.UnifiedLog$) [main]
2025-06-17 07:43:31,590 INFO Deleted producer state snapshot /var/lib/kafka/data-0/kafka-log0/__cluster_metadata-0/00000000000000007856.snapshot (org.apache.kafka.storage.internals.log.SnapshotFile) [main]
2025-06-17 07:43:31,590 INFO [ProducerStateManager partition=__cluster_metadata-0] Loading producer state from snapshot file 'SnapshotFile(offset=14303, file=/var/lib/kafka/data-0/kafka-log0/__cluster_metadata-0/00000000000000014303.snapshot)' (org.apache.kafka.storage.internals.log.ProducerStateManager) [main]
2025-06-17 07:43:31,591 INFO [LogLoader partition=__cluster_metadata-0, dir=/var/lib/kafka/data-0/kafka-log0] Producer state recovery took 1ms for snapshot load and 0ms for segment recovery from offset 14303 (kafka.log.UnifiedLog$) [main]
2025-06-17 07:43:31,621 INFO Initialized snapshots with IDs SortedSet(OffsetAndEpoch(offset=7856, epoch=25)) from /var/lib/kafka/data-0/kafka-log0/__cluster_metadata-0 (kafka.raft.KafkaMetadataLog$) [main]
2025-06-17 07:43:31,696 INFO [raft-expiration-reaper]: Starting (kafka.raft.TimingWheelExpirationService$ExpiredOperationReaper) [raft-expiration-reaper]
2025-06-17 07:43:31,706 INFO [RaftManager id=0] Reading KRaft snapshot and log as part of the initialization (org.apache.kafka.raft.KafkaRaftClient) [main]
2025-06-17 07:43:31,707 INFO [RaftManager id=0] Loading snapshot (OffsetAndEpoch(offset=7856, epoch=25)) since log start offset (7856) is greater than the internal listener's next offset (-1) (org.apache.kafka.raft.internals.KRaftControlRecordStateMachine) [main]
2025-06-17 07:43:31,876 INFO [RaftManager id=0] Starting voters are VoterSet(voters={4=VoterNode(voterKey=ReplicaKey(id=4, directoryId=Optional.empty), listeners=Endpoints(endpoints={ListenerName(CONTROLPLANE-9090)=-kafka-cluster-controller-4.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local/10.67.192.8:9090}), supportedKRaftVersion=SupportedVersionRange[min_version:0, max_version:0]), 5=VoterNode(voterKey=ReplicaKey(id=5, directoryId=Optional.empty), listeners=Endpoints(endpoints={ListenerName(CONTROLPLANE-9090)=-kafka-cluster-controller-5.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local/10.67.194.14:9090}), supportedKRaftVersion=SupportedVersionRange[min_version:0, max_version:0]), 6=VoterNode(voterKey=ReplicaKey(id=6, directoryId=Optional.empty), listeners=Endpoints(endpoints={ListenerName(CONTROLPLANE-9090)=-kafka-cluster-controller-6.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local/10.67.193.197:9090}), supportedKRaftVersion=SupportedVersionRange[min_version:0, max_version:0])}) (org.apache.kafka.raft.KafkaRaftClient) [main]
2025-06-17 07:43:31,877 INFO [RaftManager id=0] Starting request manager with static voters: [-kafka-cluster-controller-4.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local:9090 (id: 4 rack: null), -kafka-cluster-controller-5.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local:9090 (id: 5 rack: null), -kafka-cluster-controller-6.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local:9090 (id: 6 rack: null)] (org.apache.kafka.raft.KafkaRaftClient) [main]
2025-06-17 07:43:31,881 WARN [RaftManager id=0] Epoch from quorum store file (/var/lib/kafka/data-0/kafka-log0/__cluster_metadata-0/quorum-state) is 0, which is smaller than last written epoch 25 in the log (org.apache.kafka.raft.QuorumState) [main]
2025-06-17 07:43:31,882 INFO [RaftManager id=0] Attempting durable transition to Unattached(epoch=25, votedKey=null, voters=[4, 5, 6], electionTimeoutMs=3439, highWatermark=Optional.empty) from null (org.apache.kafka.raft.QuorumState) [main]
2025-06-17 07:43:32,091 INFO [RaftManager id=0] Completed transition to Unattached(epoch=25, votedKey=null, voters=[4, 5, 6], electionTimeoutMs=3439, highWatermark=Optional.empty) from null (org.apache.kafka.raft.QuorumState) [main]
2025-06-17 07:43:32,094 INFO [kafka-0-raft-outbound-request-thread]: Starting (org.apache.kafka.raft.KafkaNetworkChannel$SendThread) [kafka-0-raft-outbound-request-thread]
2025-06-17 07:43:32,094 INFO [kafka-0-raft-io-thread]: Starting (org.apache.kafka.raft.KafkaRaftClientDriver) [kafka-0-raft-io-thread]
2025-06-17 07:43:32,113 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:32,114 INFO [BrokerServer id=0] Starting broker (kafka.server.BrokerServer) [main]
2025-06-17 07:43:32,116 INFO Configuring EnvVar config provider: {} (io.strimzi.kafka.EnvVarConfigProvider) [main]
2025-06-17 07:43:32,116 INFO Closing EnvVar config provider (io.strimzi.kafka.EnvVarConfigProvider) [main]
2025-06-17 07:43:32,175 INFO [broker-0-ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [broker-0-ThrottledChannelReaper-Fetch]
2025-06-17 07:43:32,175 INFO [broker-0-ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [broker-0-ThrottledChannelReaper-Produce]
2025-06-17 07:43:32,176 INFO [broker-0-ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [broker-0-ThrottledChannelReaper-Request]
2025-06-17 07:43:32,178 INFO [broker-0-ThrottledChannelReaper-ControllerMutation]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [broker-0-ThrottledChannelReaper-ControllerMutation]
2025-06-17 07:43:32,269 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:32,370 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:32,394 INFO RemoteIndexCache directory /var/lib/kafka/data-0/kafka-log0/remote-log-index-cache already exists. Re-using the same directory. (org.apache.kafka.storage.internals.log.RemoteIndexCache) [main]
2025-06-17 07:43:32,394 INFO RemoteIndexCache starts up in 1 ms. (org.apache.kafka.storage.internals.log.RemoteIndexCache) [main]
2025-06-17 07:43:32,395 INFO [remote-log-index-cleaner]: Starting (org.apache.kafka.storage.internals.log.RemoteIndexCache$1) [remote-log-index-cleaner]
2025-06-17 07:43:32,468 INFO [BrokerServer id=0] Waiting for controller quorum voters future (kafka.server.BrokerServer) [main]
2025-06-17 07:43:32,468 INFO [BrokerServer id=0] Finished waiting for controller quorum voters future (kafka.server.BrokerServer) [main]
2025-06-17 07:43:32,470 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:32,570 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:32,586 INFO [broker-0-to-controller-forwarding-channel-manager]: Starting (kafka.server.NodeToControllerRequestThread) [broker-0-to-controller-forwarding-channel-manager]
2025-06-17 07:43:32,670 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:32,670 INFO [client-metrics-reaper]: Starting (org.apache.kafka.server.util.timer.SystemTimerReaper$Reaper) [client-metrics-reaper]
2025-06-17 07:43:32,676 INFO [RaftManager id=0] Registered the listener org.apache.kafka.image.loader.MetadataLoader@710231875 (org.apache.kafka.raft.KafkaRaftClient) [kafka-0-raft-io-thread]
2025-06-17 07:43:32,771 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:32,871 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:32,971 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:33,072 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:33,078 INFO Updated connection-accept-rate max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas) [main]
2025-06-17 07:43:33,172 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:33,272 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:33,372 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:33,473 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:33,573 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:33,673 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:33,773 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:33,871 INFO [RaftManager id=0] Attempting durable transition to FollowerState(fetchTimeoutMs=2000, epoch=25, leader=4, leaderEndpoints=Endpoints(endpoints={ListenerName(CONTROLPLANE-9090)=-kafka-cluster-controller-4.-kafka-cluster-kafka-brokers.-kafka.svc/<unresolved>:9090}), voters=[4, 5, 6], highWatermark=Optional.empty, fetchingSnapshot=Optional.empty) from Unattached(epoch=25, votedKey=null, voters=[4, 5, 6], electionTimeoutMs=3439, highWatermark=Optional.empty) (org.apache.kafka.raft.QuorumState) [kafka-0-raft-io-thread]
2025-06-17 07:43:33,874 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:33,896 INFO [RaftManager id=0] Completed transition to FollowerState(fetchTimeoutMs=2000, epoch=25, leader=4, leaderEndpoints=Endpoints(endpoints={ListenerName(CONTROLPLANE-9090)=-kafka-cluster-controller-4.-kafka-cluster-kafka-brokers.-kafka.svc/<unresolved>:9090}), voters=[4, 5, 6], highWatermark=Optional.empty, fetchingSnapshot=Optional.empty) from Unattached(epoch=25, votedKey=null, voters=[4, 5, 6], electionTimeoutMs=3439, highWatermark=Optional.empty) (org.apache.kafka.raft.QuorumState) [kafka-0-raft-io-thread]
2025-06-17 07:43:33,974 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:33,992 INFO [broker-0-to-controller-forwarding-channel-manager]: Recorded new KRaft controller, from now on will use node -kafka-cluster-controller-4.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local:9090 (id: 4 rack: null) (kafka.server.NodeToControllerRequestThread) [broker-0-to-controller-forwarding-channel-manager]
2025-06-17 07:43:34,074 INFO [MetadataLoader id=0] initializeNewPublishers: the loader is still catching up because we still don't know the high water mark yet. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:34,082 INFO [RaftManager id=0] High watermark set to Optional[LogOffsetMetadata(offset=14942, metadata=Optional.empty)] for the first time for epoch 25 (org.apache.kafka.raft.FollowerState) [kafka-0-raft-io-thread]
2025-06-17 07:43:34,084 INFO [MetadataLoader id=0] handleLoadSnapshot(00000000000000007856-0000000025): incrementing HandleLoadSnapshotCount to 1. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:34,172 INFO [MetadataLoader id=0] handleLoadSnapshot(00000000000000007856-0000000025): generated a metadata delta between offset -1 and this snapshot in 88483 us. (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:34,173 INFO [MetadataLoader id=0] maybePublishMetadata(SNAPSHOT): The loader is still catching up because we have loaded up to offset 7855, but the high water mark is 14942 (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:34,175 INFO [SocketServer listenerType=BROKER, nodeId=0] Created data-plane acceptor and processors for endpoint : ListenerName(REPLICATION-9091) (kafka.network.SocketServer) [main]
2025-06-17 07:43:34,175 INFO Updated connection-accept-rate max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas) [main]
2025-06-17 07:43:34,178 INFO [SocketServer listenerType=BROKER, nodeId=0] Created data-plane acceptor and processors for endpoint : ListenerName(PLAIN-9092) (kafka.network.SocketServer) [main]
2025-06-17 07:43:34,178 INFO Updated connection-accept-rate max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas) [main]
2025-06-17 07:43:34,189 INFO Successfully logged in. (org.apache.kafka.common.security.authenticator.AbstractLogin) [main]
2025-06-17 07:43:34,276 INFO [SocketServer listenerType=BROKER, nodeId=0] Created data-plane acceptor and processors for endpoint : ListenerName(TLS-9093) (kafka.network.SocketServer) [main]
2025-06-17 07:43:34,289 INFO [broker-0-to-controller-alter-partition-channel-manager]: Starting (kafka.server.NodeToControllerRequestThread) [broker-0-to-controller-alter-partition-channel-manager]
2025-06-17 07:43:34,289 INFO [broker-0-to-controller-alter-partition-channel-manager]: Recorded new KRaft controller, from now on will use node -kafka-cluster-controller-4.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local:9090 (id: 4 rack: null) (kafka.server.NodeToControllerRequestThread) [broker-0-to-controller-alter-partition-channel-manager]
2025-06-17 07:43:34,371 INFO [MetadataLoader id=0] maybePublishMetadata(LOG_DELTA): The loader finished catching up to the current high water mark of 14942 (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:34,375 INFO [MetadataLoader id=0] InitializeNewPublishers: initializing SnapshotGenerator with a snapshot at offset 14941 (org.apache.kafka.image.loader.MetadataLoader) [kafka-0-metadata-loader-event-handler]
2025-06-17 07:43:34,388 INFO [broker-0-to-controller-directory-assignments-channel-manager]: Starting (kafka.server.NodeToControllerRequestThread) [broker-0-to-controller-directory-assignments-channel-manager]
2025-06-17 07:43:34,388 INFO [broker-0-to-controller-directory-assignments-channel-manager]: Recorded new KRaft controller, from now on will use node -kafka-cluster-controller-4.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local:9090 (id: 4 rack: null) (kafka.server.NodeToControllerRequestThread) [broker-0-to-controller-directory-assignments-channel-manager]
2025-06-17 07:43:34,466 INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-Produce]
2025-06-17 07:43:34,468 INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-Fetch]
2025-06-17 07:43:34,468 INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-DeleteRecords]
2025-06-17 07:43:34,469 INFO [ExpirationReaper-0-ElectLeader]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-ElectLeader]
2025-06-17 07:43:34,470 INFO [ExpirationReaper-0-RemoteFetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-RemoteFetch]
2025-06-17 07:43:34,485 INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-Heartbeat]
2025-06-17 07:43:34,485 INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-Rebalance]
2025-06-17 07:43:34,586 INFO Unable to read the broker epoch in /var/lib/kafka/data-0/kafka-log0. (kafka.log.LogManager) [main]
2025-06-17 07:43:34,586 INFO [broker-0-to-controller-heartbeat-channel-manager]: Starting (kafka.server.NodeToControllerRequestThread) [broker-0-to-controller-heartbeat-channel-manager]
2025-06-17 07:43:34,587 INFO [broker-0-to-controller-heartbeat-channel-manager]: Recorded new KRaft controller, from now on will use node -kafka-cluster-controller-4.-kafka-cluster-kafka-brokers.-kafka.svc.cluster.local:9090 (id: 4 rack: null) (kafka.server.NodeToControllerRequestThread) [broker-0-to-controller-heartbeat-channel-manager]
2025-06-17 07:43:34,590 INFO [BrokerLifecycleManager id=0] Incarnation CN6v319-ReSimRYzngFE8g of broker 0 in cluster 7wDfh2TwQPayDWMrH1i0UQ is now STARTING. (kafka.server.BrokerLifecycleManager) [broker-0-lifecycle-manager-event-handler]
2025-06-17 07:43:34,593 INFO [StandardAuthorizer 0] set super.users=User:CN=-kafka-cluster-kafka-exporter,O=io.strimzi,User:CN=-kafka-cluster-entity-topic-operator,O=io.strimzi,User:CN=cluster-operator,O=io.strimzi,User:CN=-kafka-cluster-entity-user-operator,O=io.strimzi,User:CN=-kafka-cluster-cruise-control,O=io.strimzi,User:CN=-kafka-cluster-kafka,O=io.strimzi, default result=DENIED (org.apache.kafka.metadata.authorizer.StandardAuthorizerData) [main]
2025-06-17 07:43:34,679 INFO [ExpirationReaper-0-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [ExpirationReaper-0-AlterAcls]
2025-06-17 07:43:34,768 INFO [BrokerLifecycleManager id=0] Successfully registered broker 0 with broker epoch 14943 (kafka.server.BrokerLifecycleManager) [broker-0-lifecycle-manager-event-handler]
2025-06-17 07:43:34,770 INFO RemoteStorageManagerConfig values:
	chunk.size = 5242880
	compression.enabled = false
	compression.heuristic.enabled = false
	custom.metadata.fields.include = []
	encryption.enabled = false
	key.prefix =
	key.prefix.mask = false
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	storage.backend.class = class io.aiven.kafka.tieredstorage.storage.azure.AzureBlobStorage
	upload.rate.limit.bytes.per.second = null
 (io.aiven.kafka.tieredstorage.config.RemoteStorageManagerConfig) [main]
2025-06-17 07:43:34,775 INFO AzureBlobStorageConfig values:
	azure.account.key = null
	azure.account.name = null
	azure.connection.string = [hidden]
	azure.container.name = -devddc-kafka-strorage
	azure.endpoint.url = null
	azure.sas.token = null
	azure.upload.block.size = 26214400
 (io.aiven.kafka.tieredstorage.storage.azure.AzureBlobStorageConfig) [main]
2025-06-17 07:43:34,792 ERROR Invalid connection string. (com.azure.storage.blob.BlobServiceClientBuilder) [main]
2025-06-17 07:43:34,792 INFO [BrokerServer id=0] Transition from STARTING to STARTED (kafka.server.BrokerServer) [main]
2025-06-17 07:43:34,794 ERROR [BrokerServer id=0] Fatal error during broker startup. Prepare to shutdown (kafka.server.BrokerServer) [main]
java.lang.IllegalArgumentException: Invalid connection string.
	at com.azure.storage.common.implementation.connectionstring.ConnectionSettings.fromConnectionString(ConnectionSettings.java:81)
	at com.azure.storage.common.implementation.connectionstring.StorageConnectionString.create(StorageConnectionString.java:105)
	at com.azure.storage.blob.BlobServiceClientBuilder.connectionString(BlobServiceClientBuilder.java:323)
	at io.aiven.kafka.tieredstorage.storage.azure.AzureBlobStorage.configure(AzureBlobStorage.java:60)
	at io.aiven.kafka.tieredstorage.config.RemoteStorageManagerConfig.storage(RemoteStorageManagerConfig.java:318)
	at io.aiven.kafka.tieredstorage.RemoteStorageManager.configure(RemoteStorageManager.java:151)
	at org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager.lambda$configure$0(ClassLoaderAwareRemoteStorageManager.java:48)
	at org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager.withClassLoader(ClassLoaderAwareRemoteStorageManager.java:65)
	at org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager.configure(ClassLoaderAwareRemoteStorageManager.java:47)
	at kafka.log.remote.RemoteLogManager.configureRSM(RemoteLogManager.java:353)
	at kafka.log.remote.RemoteLogManager.startup(RemoteLogManager.java:396)
	at kafka.server.BrokerServer.$anonfun$startup$18(BrokerServer.scala:438)
	at kafka.server.BrokerServer.$anonfun$startup$18$adapted(BrokerServer.scala:425)
	at scala.Option.foreach(Option.scala:437)
	at kafka.server.BrokerServer.startup(BrokerServer.scala:425)
	at kafka.server.KafkaRaftServer.$anonfun$startup$2(KafkaRaftServer.scala:99)
	at kafka.server.KafkaRaftServer.$anonfun$startup$2$adapted(KafkaRaftServer.scala:99)
	at scala.Option.foreach(Option.scala:437)
	at kafka.server.KafkaRaftServer.startup(KafkaRaftServer.scala:99)
	at kafka.Kafka$.main(Kafka.scala:112)
	at kafka.Kafka.main(Kafka.scala)
2025-06-17 07:43:34,796 INFO [BrokerServer id=0] Transition from STARTED to SHUTTING_DOWN (kafka.server.BrokerServer) [main]
2025-06-17 07:43:34,796 INFO [BrokerServer id=0] shutting down (kafka.server.BrokerServer) [main]

And the broker config:

---
# Source: strimzi-kafka-operator/templates/kafka-node-pool.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: broker
  labels:
    strimzi.io/cluster: kafka-cluster
spec:
  replicas: 4
  resources:
    requests:
      memory: "5Gi"
      cpu: "700m"
    limits:
      memory: "6Gi"
      cpu: "1000m"
  roles:
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: "50Gi"
        kraftMetadata: shared
        deleteClaim: false
  template:
    pod:
      metadata:
        annotations:
          prometheus.io/path: /metrics
          prometheus.io/port: "9404"
          prometheus.io/scrape: "true"
    kafkaContainer:
      env:
        - name: AZURE_CONNECTION_STRING
          valueFrom:
            secretKeyRef:
              name: "kafka-strimzi-kafka-operator-vault-secrets"
              key: "DEV_KAFKA_STORAGE_CONNECTION_STRING"

[scholzj]

Well, the exception comes from the tiered storage plugin. So you should maybe direct your question there. But I guess you will still need the full log from your cluster. It also xould be the problem is related to how cou configure it - maybe you are missing some configuration. Or if it is configured by ${env:AZURE_CONNECTION_STRING} it is not correctly interpreted.