Why are my secrets being deleted? #8724

rjones-duchancell · 2023-06-23T23:35:08Z

rjones-duchancell
Jun 23, 2023

I am attempting to use custom certificates in my deployment and have set generateCertificateAuthority false for clientsCa. I have followed the documentation and have labeled and annotated the secrets appropriately. The secrets are created, but after a few minutes they are deleted. I am deploying to OpenShift. What behavior is causing this? Kafka and Zookeeper pods deploy correctly, but the Entity Operator is failing with an error message saying it can't find the wds-cluster-ca-cert. The Strimzi pod looks good initially, then starts reporting errors not being able to find certs.

Is there some garbage collection going on behind the scenes that I need to configure or turn off?

BTW, I did add the external listener cert configuration after reading (and commenting on) another discussion. Is there a conflict going on?

Here is my template:

apiVersion: template.openshift.io/v1
kind: Template
metadata:
  name: wds-kafka-aa-template
  annotations:
    description: "Template used to deploy Kafka via AA"
    tags: "wds,kafka"
labels:
  template: wds-kafka-aa-template
objects:

- apiVersion: v1
    kind: Secret
    type: Opaque
    metadata:
      name: wds-clients-ca
      annotations:
        strimzi.io/ca-key-generation: "0"
      labels:
        strimzi.io/kind: Kafka
        strimzi.io/cluster: wds
    data:
      ca.key: ${TLS_KEY}
    
  - apiVersion: v1
    kind: Secret
    type: Opaque
    metadata:
      name: wds-clients-ca-cert
      annotations:
        strimzi.io/ca-cert-generation: "0"
      labels:
        strimzi.io/kind: Kafka
        strimzi.io/cluster: wds
    data:
      ca.crt: ${TLS_CERT}
      
  - apiVersion: v1
    kind: Secret
    type: kubernetes.io/tls
    metadata:
      name: wds-cert-chain
    data:
      tls.crt: ${TLS_CERT}
      tls.key: ${TLS_KEY}      

  - apiVersion: kafka.strimzi.io/v1beta2
    kind: Kafka
    metadata:
      name: wds
    spec:
      clientsCa:
        generateCertificateAuthority: false
      kafka:
        replicas: 1
        version: 3.4.0
        image: ${AA_REGISTRY}/wds/kafka:0.34.0-kafka-3.4.0
        resources:
          requests:
            memory: 4Gi
            cpu: "1"
          limits:
            memory: 4Gi
            cpu: "1"
        readinessProbe:
          initialDelaySeconds: 15
          timeoutSeconds: 5
        livenessProbe:
          initialDelaySeconds: 15
          timeoutSeconds: 5
        jvmOptions:
          -Xms: 2048m
          -Xmx: 2048m
        listeners:
          - name: plain
            port: 9092
            type: internal
            tls: false
            configuration:
              useServiceDnsDomain: true
          - name: tls
            port: 9093
            type: internal
            tls: true
            authentication:
              type: tls
          - name: external
            port: 9094
            type: route
            tls: true
            configuration:
              brokerCertChainAndKey:
                secretName: wds-cert-chain
                certificate: tls.crt
                key: tls.key
        config:
          offsets.topic.replication.factor: 1
          transaction.state.log.replication.factor: 1
          transaction.state.log.min.isr: 1
          default.replication.factor: 1
          min.insync.replicas: 1
          inter.broker.protocol.version: "3.4"
          message.max.bytes: 104857600
        storage:
          type: persistent-claim
          size: ${KAFKA_STORAGE_SIZE}
          class: ${STORAGE_CLASS_NAME}
          deleteClaim: false
        logging:
          loggers:
            kafka.root.logger.level: ${KAFKA_ROOT_LOGGER_LEVEL}
          type: inline  
        template:
          pod:
            imagePullSecrets:
              - name: aa-dockerconfig
      zookeeper:
        image: ${AA_REGISTRY}/wds/kafka:0.34.0-kafka-3.4.0
        replicas: 1
        storage:
          type: persistent-claim
          size: ${ZOOKEEPER_STORAGE_SIZE}
          class: ${STORAGE_CLASS_NAME}
          deleteClaim: false
        logging:
          loggers:
            zookeeper.root.logger: ${ZOOKEEPER_ROOT_LOGGER_LEVEL}
          type: inline      
        template:
          pod:
            imagePullSecrets:
              - name: aa-dockerconfig
      entityOperator:
        topicOperator: {}
        userOperator: {}             

parameters:

  - name: AA_REGISTRY
    description: "App Arsenal Docker registry (environment-specific)"
    value: acs-arsenal.apps.communal3.sd.niwc.navy.mil
    required: true
    
  - name: NAMESPACE
    description: "OpenShift namespace"
    value: wds-kafka
    required: true  
    
  - name: STORAGE_CLASS_NAME
    description: "The storage class to use for persistent volume claims"
    value: gp2
    required: true  
    
  - name: KAFKA_STORAGE_SIZE
    description: "Persistent storage request for Kafka"
    value: 100Gi
    required: true
    
  - name: KAFKA_ROOT_LOGGER_LEVEL
    description: "Root logger level for Kafka"
    value: INFO
    required: true    
    
  - name: ZOOKEEPER_STORAGE_SIZE
    description: "Persistent storage request for Zookeeper"
    value: 25Gi
    required: true
    
  - name: ZOOKEEPER_ROOT_LOGGER_LEVEL
    description: "Root logger level for Zookeeper"
    value: INFO
    required: true
    
  - name: TLS_KEY
    description: "Server Private Key (Change Per Deployment Environment)"
    value: <encoded private key>
    required: true
    
  - name: TLS_CERT
    description: "Server Certificate (Change Per Deployment Environment)"
    value: <encoded certificate with trust chain, 3 certs total>
    required: true

scholzj · 2023-06-24T12:23:37Z

scholzj
Jun 24, 2023
Maintainer

You should probably share the logs from the operator and from the failing pods to help us better understand the issue.

14 replies

rjones-duchancell Jun 24, 2023
Author

Ok, I'll fix that. But does that explain the deleted secrets? If all containers don't come up healthy, does the system try to reconcile back to some known state?

scholzj Jun 24, 2023
Maintainer

It does not explain any deleted secrets.

rjones-duchancell Jun 27, 2023
Author

Jakub, any more thought on this? I think my deployment is fairly simple, maybe not super common with the custom certs.

scholzj Jun 27, 2023
Maintainer

TBH, none of the logs you shared show anything super useful. The operator log shows that when deploying the Kafka cluster, it correctly generated only the Cluster CA (that is correct since you provided the Clients CA). Then it waits for the Entity OPerator deployment -> I assume that did not deploy because of the image pull secret missing. And then the Clients CA is missing.

My best guess is that some other tooling deleted it. Hence the question about using Argo or other GitOps tooling since they might do it. If you repeat it with DEBUG logging, you should be able to see if the operator deletes the secret.

This setup is not that rare, so you are definitely not the only one using it. But I'm not aware any other issues.

rjones-duchancell Jun 27, 2023
Author

I can't think of anything that could be deleting them. There are a bunch of projects in the OpenShift cluster. As far as I know, Argo is not deployed in the environment. I will check with the cluster admin to see if anyone else has had any similar behavior, but I'd be surprised.

rjones-duchancell · 2023-06-24T19:31:35Z

rjones-duchancell
Jun 24, 2023
Author

Entire entity operator log:

removed directory '/tmp/hsperfdata_1001980000'
removed '/tmp/topic-operator/replication.keystore.p12'
removed '/tmp/topic-operator/replication.truststore.p12'
removed directory '/tmp/topic-operator'
Preparing trust store certificates for internal communication
Adding /etc/tls-sidecar/cluster-ca-certs/ca.crt to truststore /tmp/topic-operator/replication.truststore.p12 with alias ca
Certificate was added to keystore
Preparing trust store certificates for internal communication is completed
Preparing key store certificates for internal communication
Preparing key store certificates for internal communication is completed
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/topic-operator/custom-config/log4j2.properties -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=15 -XX:MaxRAMPercentage=20 -classpath :lib/io.strimzi.topic-operator-0.34.0.jar:lib/io.strimzi.api-0.34.0.jar:lib/io.fabric8.kubernetes-model-networking-6.3.1.jar:lib/io.fabric8.kubernetes-model-common-6.3.1.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.3.1.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.14.1.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.14.1.jar:lib/io.fabric8.kubernetes-client-6.3.1.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.fabric8.kubernetes-httpclient-jdk-6.3.1.jar:lib/io.fabric8.kubernetes-client-api-6.3.1.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.3.1.jar:lib/io.fabric8.kubernetes-model-rbac-6.3.1.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.3.1.jar:lib/io.fabric8.kubernetes-model-apps-6.3.1.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.3.1.jar:lib/io.fabric8.kubernetes-model-batch-6.3.1.jar:lib/io.fabric8.kubernetes-model-certificates-6.3.1.jar:lib/io.fabric8.kubernetes-model-coordination-6.3.1.jar:lib/io.fabric8.kubernetes-model-discovery-6.3.1.jar:lib/io.fabric8.kubernetes-model-events-6.3.1.jar:lib/io.fabric8.kubernetes-model-extensions-6.3.1.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.3.1.jar:lib/io.fabric8.kubernetes-model-metrics-6.3.1.jar:lib/io.fabric8.kubernetes-model-policy-6.3.1.jar:lib/io.fabric8.kubernetes-model-scheduling-6.3.1.jar:lib/io.fabric8.kubernetes-model-storageclass-6.3.1.jar:lib/io.fabric8.kubernetes-model-node-6.3.1.jar:lib/org.yaml.snakeyaml-1.33.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.14.1.jar:lib/io.fabric8.kubernetes-model-core-6.3.1.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/org.apache.logging.log4j.log4j-jul-2.17.2.jar:lib/org.apache.kafka.kafka-clients-3.4.0.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-handler-4.1.87.Final.jar:lib/io.netty.netty-transport-native-unix-common-4.1.87.Final.jar:lib/io.netty.netty-codec-4.1.87.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.87.Final.jar:lib/io.netty.netty-transport-classes-epoll-4.1.87.Final.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/com.fasterxml.jackson.core.jackson-core-2.14.1.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.14.1.jar:lib/io.vertx.vertx-core-4.3.8.jar:lib/io.netty.netty-common-4.1.87.Final.jar:lib/io.netty.netty-buffer-4.1.87.Final.jar:lib/io.netty.netty-transport-4.1.87.Final.jar:lib/io.netty.netty-handler-proxy-4.1.87.Final.jar:lib/io.netty.netty-codec-socks-4.1.87.Final.jar:lib/io.netty.netty-codec-http-4.1.87.Final.jar:lib/io.netty.netty-codec-http2-4.1.87.Final.jar:lib/io.netty.netty-resolver-4.1.87.Final.jar:lib/io.netty.netty-resolver-dns-4.1.87.Final.jar:lib/io.netty.netty-codec-dns-4.1.87.Final.jar:lib/io.apicurio.apicurio-registry-utils-streams-1.3.2.Final.jar:lib/jakarta.annotation.jakarta.annotation-api-1.3.5.jar:lib/io.apicurio.apicurio-registry-utils-kafka-1.3.2.Final.jar:lib/org.eclipse.microprofile.config.microprofile-config-api-1.4.jar:lib/org.apache.kafka.kafka-streams-3.4.0.jar:lib/org.rocksdb.rocksdbjni-7.1.2.jar:lib/io.fabric8.openshift-client-api-6.3.1.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.3.1.jar:lib/io.fabric8.openshift-model-operator-6.3.1.jar:lib/io.fabric8.openshift-model-machine-6.3.1.jar:lib/io.fabric8.openshift-model-whereabouts-6.3.1.jar:lib/io.fabric8.openshift-model-monitoring-6.3.1.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.3.1.jar:lib/io.fabric8.openshift-model-tuned-6.3.1.jar:lib/io.fabric8.openshift-model-console-6.3.1.jar:lib/io.fabric8.openshift-model-config-6.3.1.jar:lib/io.fabric8.openshift-model-machineconfig-6.3.1.jar:lib/io.fabric8.openshift-model-miscellaneous-6.3.1.jar:lib/io.fabric8.openshift-model-hive-6.3.1.jar:lib/io.fabric8.openshift-model-installer-6.3.1.jar:lib/io.fabric8.openshift-model-operatorhub-6.3.1.jar:lib/io.strimzi.operator-common-0.34.0.jar:lib/io.strimzi.crd-annotations-0.34.0.jar:lib/io.strimzi.certificate-manager-0.34.0.jar:lib/io.fabric8.openshift-model-6.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.micrometer.micrometer-core-1.9.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.12.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.micrometer.micrometer-registry-prometheus-1.9.5.jar:lib/io.prometheus.simpleclient_common-0.15.0.jar:lib/io.prometheus.simpleclient-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_common-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel_agent-0.15.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.3.8.jar io.strimzi.operator.topic.Main
2023-06-24 19:28:34,26375 INFO  [main] Main:38 - TopicOperator 0.34.0 is starting
2023-06-24 19:28:35,34107 INFO  [main] Session:121 - Using config:
	STRIMZI_TRUSTSTORE_LOCATION: /tmp/topic-operator/replication.truststore.p12
	STRIMZI_RESOURCE_LABELS: strimzi.io/cluster=wds
	STRIMZI_SASL_MECHANISM: 
	STRIMZI_SASL_USERNAME: 
	STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
	STRIMZI_CLIENT_ID: strimzi-topic-operator-9d5d38ca-0bea-4522-9fff-a001b8df1a1a
	STRIMZI_SECURITY_PROTOCOL: SSL
	STRIMZI_STALE_RESULT_TIMEOUT_MS: 5000
	STRIMZI_TOPIC_METADATA_MAX_ATTEMPTS: 6
	STRIMZI_KEYSTORE_LOCATION: /tmp/topic-operator/replication.keystore.p12
	STRIMZI_REASSIGN_THROTTLE: 9223372036854775807
	STRIMZI_USE_ZOOKEEPER_TOPIC_STORE: false
	STRIMZI_SASL_PASSWORD: ********
	STRIMZI_KAFKA_BOOTSTRAP_SERVERS: wds-kafka-bootstrap:9091
	STRIMZI_SASL_ENABLED: false
	STRIMZI_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: HTTPS
	STRIMZI_NAMESPACE: wds-kafka
	STRIMZI_APPLICATION_ID: __strimzi-topic-operator-kstreams
	STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS: 18000
	STRIMZI_TOPICS_PATH: /strimzi/topics
	STRIMZI_ZOOKEEPER_CONNECT: localhost:2181
	STRIMZI_TLS_ENABLED: true
	STRIMZI_KEYSTORE_PASSWORD: ********
	STRIMZI_STORE_NAME: topic-store
	STRIMZI_REASSIGN_VERIFY_INTERVAL_MS: 120000
	TC_ZK_CONNECTION_TIMEOUT_MS: 18000
	STRIMZI_TRUSTSTORE_PASSWORD: ********
	STRIMZI_STORE_TOPIC: __strimzi_store_topic

2023-06-24 19:28:35,35514 INFO  [vert.x-eventloop-thread-0] Session:209 - Starting
2023-06-24 19:28:35,49708 INFO  [vert.x-eventloop-thread-0] AppInfoParser:119 - Kafka version: 3.4.0
2023-06-24 19:28:35,49741 INFO  [vert.x-eventloop-thread-0] AppInfoParser:120 - Kafka commitId: 2e1947d240607d53
2023-06-24 19:28:35,49763 INFO  [vert.x-eventloop-thread-0] AppInfoParser:121 - Kafka startTimeMs: 1687634915495
2023-06-24 19:28:35,64933 INFO  [ZkClient-EventThread-27-localhost:2181] ZkEventThread:65 - Starting ZkClient event thread.
2023-06-24 19:28:35,65242 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT
2023-06-24 19:28:35,65288 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:host.name=wds-entity-operator-5cccf84474-r9hnv
2023-06-24 19:28:35,65310 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.version=17.0.6
2023-06-24 19:28:35,65332 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.vendor=Red Hat, Inc.
2023-06-24 19:28:35,65349 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.home=/usr/lib/jvm/java-17-openjdk-17.0.6.0.10-3.el8_7.x86_64
2023-06-24 19:28:35,65369 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.class.path=:lib/io.strimzi.topic-operator-0.34.0.jar:lib/io.strimzi.api-0.34.0.jar:lib/io.fabric8.kubernetes-model-networking-6.3.1.jar:lib/io.fabric8.kubernetes-model-common-6.3.1.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.3.1.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.14.1.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.14.1.jar:lib/io.fabric8.kubernetes-client-6.3.1.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.fabric8.kubernetes-httpclient-jdk-6.3.1.jar:lib/io.fabric8.kubernetes-client-api-6.3.1.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.3.1.jar:lib/io.fabric8.kubernetes-model-rbac-6.3.1.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.3.1.jar:lib/io.fabric8.kubernetes-model-apps-6.3.1.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.3.1.jar:lib/io.fabric8.kubernetes-model-batch-6.3.1.jar:lib/io.fabric8.kubernetes-model-certificates-6.3.1.jar:lib/io.fabric8.kubernetes-model-coordination-6.3.1.jar:lib/io.fabric8.kubernetes-model-discovery-6.3.1.jar:lib/io.fabric8.kubernetes-model-events-6.3.1.jar:lib/io.fabric8.kubernetes-model-extensions-6.3.1.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.3.1.jar:lib/io.fabric8.kubernetes-model-metrics-6.3.1.jar:lib/io.fabric8.kubernetes-model-policy-6.3.1.jar:lib/io.fabric8.kubernetes-model-scheduling-6.3.1.jar:lib/io.fabric8.kubernetes-model-storageclass-6.3.1.jar:lib/io.fabric8.kubernetes-model-node-6.3.1.jar:lib/org.yaml.snakeyaml-1.33.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.14.1.jar:lib/io.fabric8.kubernetes-model-core-6.3.1.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/org.apache.logging.log4j.log4j-jul-2.17.2.jar:lib/org.apache.kafka.kafka-clients-3.4.0.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-handler-4.1.87.Final.jar:lib/io.netty.netty-transport-native-unix-common-4.1.87.Final.jar:lib/io.netty.netty-codec-4.1.87.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.87.Final.jar:lib/io.netty.netty-transport-classes-epoll-4.1.87.Final.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/com.fasterxml.jackson.core.jackson-core-2.14.1.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.14.1.jar:lib/io.vertx.vertx-core-4.3.8.jar:lib/io.netty.netty-common-4.1.87.Final.jar:lib/io.netty.netty-buffer-4.1.87.Final.jar:lib/io.netty.netty-transport-4.1.87.Final.jar:lib/io.netty.netty-handler-proxy-4.1.87.Final.jar:lib/io.netty.netty-codec-socks-4.1.87.Final.jar:lib/io.netty.netty-codec-http-4.1.87.Final.jar:lib/io.netty.netty-codec-http2-4.1.87.Final.jar:lib/io.netty.netty-resolver-4.1.87.Final.jar:lib/io.netty.netty-resolver-dns-4.1.87.Final.jar:lib/io.netty.netty-codec-dns-4.1.87.Final.jar:lib/io.apicurio.apicurio-registry-utils-streams-1.3.2.Final.jar:lib/jakarta.annotation.jakarta.annotation-api-1.3.5.jar:lib/io.apicurio.apicurio-registry-utils-kafka-1.3.2.Final.jar:lib/org.eclipse.microprofile.config.microprofile-config-api-1.4.jar:lib/org.apache.kafka.kafka-streams-3.4.0.jar:lib/org.rocksdb.rocksdbjni-7.1.2.jar:lib/io.fabric8.openshift-client-api-6.3.1.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.3.1.jar:lib/io.fabric8.openshift-model-operator-6.3.1.jar:lib/io.fabric8.openshift-model-machine-6.3.1.jar:lib/io.fabric8.openshift-model-whereabouts-6.3.1.jar:lib/io.fabric8.openshift-model-monitoring-6.3.1.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.3.1.jar:lib/io.fabric8.openshift-model-tuned-6.3.1.jar:lib/io.fabric8.openshift-model-console-6.3.1.jar:lib/io.fabric8.openshift-model-config-6.3.1.jar:lib/io.fabric8.openshift-model-machineconfig-6.3.1.jar:lib/io.fabric8.openshift-model-miscellaneous-6.3.1.jar:lib/io.fabric8.openshift-model-hive-6.3.1.jar:lib/io.fabric8.openshift-model-installer-6.3.1.jar:lib/io.fabric8.openshift-model-operatorhub-6.3.1.jar:lib/io.strimzi.operator-common-0.34.0.jar:lib/io.strimzi.crd-annotations-0.34.0.jar:lib/io.strimzi.certificate-manager-0.34.0.jar:lib/io.fabric8.openshift-model-6.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.micrometer.micrometer-core-1.9.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.12.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.micrometer.micrometer-registry-prometheus-1.9.5.jar:lib/io.prometheus.simpleclient_common-0.15.0.jar:lib/io.prometheus.simpleclient-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_common-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel_agent-0.15.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.3.8.jar
2023-06-24 19:28:35,65422 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2023-06-24 19:28:35,65443 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.io.tmpdir=/tmp
2023-06-24 19:28:35,65462 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.compiler=<NA>
2023-06-24 19:28:35,65481 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.name=Linux
2023-06-24 19:28:35,65499 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.arch=amd64
2023-06-24 19:28:35,65516 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.version=4.18.0-305.65.1.el8_4.x86_64
2023-06-24 19:28:35,65535 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:user.name=1001980000
2023-06-24 19:28:35,65552 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:user.home=/
2023-06-24 19:28:35,65567 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:user.dir=/opt/strimzi
2023-06-24 19:28:35,65582 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.memory.free=133MB
2023-06-24 19:28:35,65598 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.memory.max=12520MB
2023-06-24 19:28:35,65615 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.memory.total=160MB
2023-06-24 19:28:35,65870 INFO  [vert.x-worker-thread-0] ZooKeeper:1006 - Initiating client connection, connectString=localhost:2181 sessionTimeout=18000 watcher=org.I0Itec.zkclient.ZkClient@24f3a7d2
2023-06-24 19:28:35,66299 INFO  [vert.x-worker-thread-0] X509Util:77 - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2023-06-24 19:28:35,66713 INFO  [vert.x-worker-thread-0] ClientCnxnSocket:239 - jute.maxbuffer value is 1048575 Bytes
2023-06-24 19:28:35,67502 INFO  [vert.x-worker-thread-0] ClientCnxn:1736 - zookeeper.request.timeout value is 0. feature enabled=false
2023-06-24 19:28:35,67564 INFO  [vert.x-worker-thread-0] ZkClient:936 - Waiting for keeper state SyncConnected
2023-06-24 19:28:35,67959 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181.
2023-06-24 19:28:35,67997 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:35,68076 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:36,78727 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181.
2023-06-24 19:28:36,78761 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:36,78827 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:37,88916 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181.
2023-06-24 19:28:37,88945 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:37,89002 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:38,99095 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181.
2023-06-24 19:28:38,99117 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:38,99180 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:40,09284 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-06-24 19:28:40,09310 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:40,09374 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:41,19486 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181.
2023-06-24 19:28:41,19517 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:41,19804 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:42,29929 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181.
2023-06-24 19:28:42,29965 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:42,30029 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:43,40742 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-06-24 19:28:43,41085 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:43,41158 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:44,51257 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181.
2023-06-24 19:28:44,51277 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:44,51342 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:45,61437 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-06-24 19:28:45,61461 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:45,61510 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:46,71596 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181.
2023-06-24 19:28:46,71625 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:46,71674 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:47,81761 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-06-24 19:28:47,81793 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:47,81849 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:48,91935 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-06-24 19:28:48,91956 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:48,92012 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:50,02098 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-06-24 19:28:50,02123 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:50,02178 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:51,12268 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-06-24 19:28:51,12293 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:51,12345 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:52,22425 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-06-24 19:28:52,22453 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:52,22509 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:53,32610 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/[0:0:0:0:0:0:0:1]:2181.
2023-06-24 19:28:53,32635 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:53,32683 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1300 - Session 0x0 for sever localhost/[0:0:0:0:0:0:0:1]:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:53,67558 INFO  [ZkClient-EventThread-27-localhost:2181] ZkEventThread:83 - Terminate ZkClient event thread.
2023-06-24 19:28:54,42772 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-06-24 19:28:54,42806 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-24 19:28:54,42872 WARN  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1294 - An exception was thrown while closing send thread for session 0x0.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) ~[org.apache.zookeeper.zookeeper-3.6.3.jar:3.6.3]
2023-06-24 19:28:54,53217 INFO  [vert.x-worker-thread-0] ZooKeeper:1619 - Session: 0x0 closed
2023-06-24 19:28:54,53263 INFO  [vert.x-worker-thread-0-EventThread] ClientCnxn:578 - EventThread shut down for session: 0x0
2023-06-24 19:28:54,53302 ERROR [vert.x-eventloop-thread-0] Session:278 - Failed to start ZK client
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'localhost:2181' with timeout of 18000 ms
	at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233) ~[com.101tec.zkclient-0.11.jar:?]
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157) ~[com.101tec.zkclient-0.11.jar:?]
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131) ~[com.101tec.zkclient-0.11.jar:?]
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98) ~[com.101tec.zkclient-0.11.jar:?]
	at io.strimzi.operator.topic.zk.Zk.createSync(Zk.java:53) ~[io.strimzi.topic-operator-0.34.0.jar:0.34.0]
	at io.strimzi.operator.topic.zk.Zk.lambda$create$0(Zk.java:36) ~[io.strimzi.topic-operator-0.34.0.jar:0.34.0]
	at io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-06-24 19:28:54,54060 ERROR [vert.x-eventloop-thread-0] Session:246 - Topic operator start up failed, cause was
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'localhost:2181' with timeout of 18000 ms
	at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233) ~[com.101tec.zkclient-0.11.jar:?]
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157) ~[com.101tec.zkclient-0.11.jar:?]
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131) ~[com.101tec.zkclient-0.11.jar:?]
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98) ~[com.101tec.zkclient-0.11.jar:?]
	at io.strimzi.operator.topic.zk.Zk.createSync(Zk.java:53) ~[io.strimzi.topic-operator-0.34.0.jar:0.34.0]
	at io.strimzi.operator.topic.zk.Zk.lambda$create$0(Zk.java:36) ~[io.strimzi.topic-operator-0.34.0.jar:0.34.0]
	at io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-06-24 19:28:54,54076 ERROR [vert.x-eventloop-thread-1] Main:70 - Error deploying Session
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'localhost:2181' with timeout of 18000 ms
	at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233) ~[com.101tec.zkclient-0.11.jar:?]
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157) ~[com.101tec.zkclient-0.11.jar:?]
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131) ~[com.101tec.zkclient-0.11.jar:?]
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98) ~[com.101tec.zkclient-0.11.jar:?]
	at io.strimzi.operator.topic.zk.Zk.createSync(Zk.java:53) ~[io.strimzi.topic-operator-0.34.0.jar:0.34.0]
	at io.strimzi.operator.topic.zk.Zk.lambda$create$0(Zk.java:36) ~[io.strimzi.topic-operator-0.34.0.jar:0.34.0]
	at io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-06-24 19:29:22,18563 INFO  [Thread-2] ShutdownHook:51 - Shutdown started
2023-06-24 19:29:22,18615 INFO  [Thread-2] ShutdownHook:68 - Shutdown completed

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Why are my secrets being deleted? #8724

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 14 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

Why are my secrets being deleted? #8724

Uh oh!

Uh oh!

rjones-duchancell Jun 23, 2023

Replies: 2 comments · 14 replies

Uh oh!

scholzj Jun 24, 2023 Maintainer

Uh oh!

rjones-duchancell Jun 24, 2023 Author

Uh oh!

scholzj Jun 24, 2023 Maintainer

Uh oh!

rjones-duchancell Jun 27, 2023 Author

Uh oh!

scholzj Jun 27, 2023 Maintainer

Uh oh!

rjones-duchancell Jun 27, 2023 Author

Uh oh!

rjones-duchancell Jun 24, 2023 Author

rjones-duchancell
Jun 23, 2023

Replies: 2 comments 14 replies

scholzj
Jun 24, 2023
Maintainer

rjones-duchancell Jun 24, 2023
Author

scholzj Jun 24, 2023
Maintainer

rjones-duchancell Jun 27, 2023
Author

scholzj Jun 27, 2023
Maintainer

rjones-duchancell Jun 27, 2023
Author

rjones-duchancell
Jun 24, 2023
Author