Kafka PVCs out of sync and showing pending deletion status in Argo CD

[VijayPatil872]

Bug Description

We have deployed 'kafka with kRaft', with 'strimzi operator version 0.45.0'. We are facing issue related to PVCs. The 'ArgoCD' is out of sync and PVCs are going in to progressing giving status 'pending deletion'. All the kafka broker and controllers PVC appear as out of sync in ArgoCD, as if they need to get deleted. The PVCs get created but ArgoCD says they need to be deleted. We tried some workaround as stated above also we tried some workaround related to ignoreDifferences in ArgoCD application manifests but still issue persists.

Steps to reproduce

No response

Expected behavior

PVCs should not get in pending deletion state.

Strimzi version

strimzi operator version 0.45.0

Kubernetes version

v1.31.3

Installation method

Helm chart

Infrastructure

AKS

Configuration files and logs

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
name: controller
labels:
strimzi.io/cluster: *********
spec:
replicas: 3
resources:
requests:
memory: 3Gi
cpu: 700m
limits:
memory: 4Gi
cpu: 1000m
roles:
- controller
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 30Gi
kraftMetadata: shared
deleteClaim: false
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
name: broker
labels:
strimzi.io/cluster: ************
spec:
replicas: 4
resources:
requests:
memory: 5Gi
cpu: 700m
limits:
memory: 6Gi
cpu: 1000m
roles:
- broker
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 30Gi
kraftMetadata: shared
deleteClaim: false
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: kafka-kraft-cluster
annotations:
strimzi.io/node-pools: enabled
strimzi.io/kraft: enabled
spec:
kafka:
version: 3.9.0
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
authentication:
type: scram-sha-512
authorization:
type: simple
config:
config.providers: env
config.providers.env.class: io.strimzi.kafka.EnvVarConfigProvider
auto.create.topic.enable: "false"
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
default.replication.factor: 3
min.insync.replicas: 2
message.max.bytes: 5000000 #400000000 #50485760
replica.fetch.max.bytes: 5000000 #400000000 #50485760
fetch.message.max.bytes: 5000000 #400000000 #50485760
max.request.size: 5000000 #400000000 #50485760
producer.max.request.size: 5000000 #400000000 #50485760
log.segment.bytes: 134217728 # 128 MB
log.roll.ms: 300000 # 5 minutes
log.roll.jitter.ms: 60000 # 1 minute
log.retention.ms: 43200000 #43200000=1/2day #86400000 #1days
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
name: kafka-metrics
key: kafka-metrics-config.yml
entityOperator:
topicOperator:
resources:
requests:
memory: 500Mi
cpu: 400m
limits:
memory: 500Mi
cpu: 500m
userOperator:
resources:
requests:
memory: 500Mi
cpu: 400m
limits:
memory: 500Mi
cpu: 500m
kafkaExporter:
topicRegex: "."
groupRegex: "."
resources:
requests:
memory: 500Mi
cpu: 400m
limits:
memory: 500Mi
cpu: 500m

[Frawless]

@VijayPatil872 please, format the yamls properly.

Based on my experience you can try to set prune=False (I am using it on ApplicationSet level and it works fine without that issue for my case).

[VijayPatil872]

Hi @Frawless,
Here 'ApplicationSet' level means you mean to add it in ArgoCD application yml. Is it right?

[Frawless]

hi, byApplicationSet levle I mean set it in ApplicationSet CR. I am not sure how exactly is it set on Application level. In one of the discussion you posted same question there were suggestion of using annotation to achieve that. You should be able to find it in Argo docs.