Kafka controller backup/PVC storage decrease

[TaviDorvos]

Hi! I have the following situation: I have a Kafka controller Nodepool with 3 replicas, each with 100GB. That is way too much for this controller, and I want to decrease this size, but shrinking PVC is not supported by Kubernetes. I can't delete the PVC because I am going to lose the data.
My idea is to:

• create new PVCs (pvc-0, pvc-1, pvc-2)
• set strimzi operator replica to 0, controller pods to 0
• move the data from /var/lib/kafka/data-0 from the current PVCs (data-0-kafka-kafka-0/1/2) to the new ones (pvc-0/1/2) using rsync
• after I moved the data to pvc-0/1/2, I will delete the old ones (data-0-kafka-kafka-0/1/2)
• I create manually the same PVCs with the same name (data-0-kafka-kafka-0/1/2) and move the data back to these ones from pvc-0/1/2
• start the operator, set controller replicas to 3 and Kafka will link automatically to the new PVCs with the same name (data-0-kafka-kafka-0/1/2)

I hope I explained well enough.

I just wanted to ask if it is any other way or any good practice I can backup the data from the controller PVC?

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controller-kafka
  namespace: {{ .Release.Namespace }}
spec:
  replicas: 3
  roles:
    - controller
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false

Strimzi Version: 0.46.1
Installation: Helm Chart with ArgoCD

[scholzj]

As you are using JBOD storage, the easiest way to do this would be to:

• Add a new smaller volume to the JBOD array
• Wait for it to be added (rolling update of all controller nodes)
• Use the kraftMetadata flag to move the KRaft metadata log to the new volume
• Once it is moved (another rolling update), remove the odl volume

[TaviDorvos]

That helps me a lot, I didn't know about that flag. It makes everything much easier.

Thank you very much!