Kafka pvcs out of sync in Argo

[popdaniel942]

Starting with operator 0.29.0 all the kafka pvcs appear as out of sync in Argo, as if they need to get deleted. Trying to force sync or something still doesn't work. The pvcs get created but Argo says they need to be deleted. This didn't happen to previous versions of the operator. Is this normal behaviour? Did the creation of pvcs change?

[scholzj]

0.30.0 changed the default from StatefulSets to StrimziPodSets (there are IIRC other discussions on this on Slack and here on GitHub). But there was no change in 0.29.

[AydinChavez]

I think I have to set Argo to ignore pvcs.
@popdaniel942 How did you manage this?

like this maybe?

   kafka:
      template:
        persistentVolumeClaim:
          metadata:
            annotations:
              argocd.argoproj.io/sync-options: Prune=false

[popdaniel942]

You can't tell Argo to ignore pvcs unfortuantely. If you have problems with Argo seeing the pvcs as out of sync in argo, you can either use a featureGate (-UseStrimziPodSet) in the Strimzi operator and it will revert to uisng StatefulSets instead of StrimziPodSets and the pvcs will no longer be out of sync. Or there is a setting for when you deploy argo https://argo-cd.readthedocs.io/en/latest/user-guide/resource_tracking/ . You can use application.resourceTrackingMethod: annotation or application.resourceTrackingMethod: annotation+label and argo will know better what resources actually need to be managed by Argo. I use this one and argo no longer sees the kafka pvcs

[popdaniel942]

This same thing was talked about here:
#7093

[AydinChavez]

I am using following annotation for ignoring the pvcs created by strimzi in argocd sync state now (along with prune=false annotation):

https://argo-cd.readthedocs.io/en/stable/user-guide/compare-options/

[scholzj]

Please keep in mind that the feature gate will not be there forever. So while short term it can help, it is not a long term solution. So if you use Argo, it would be good if you can figure out how does it track the PVCs with StatefulSets and if the same mechanism can be used for StrimziPodSets / Pods or if this is completely custom mechanism which works for StatefulSets only. Because for StatefulSets it's IMHO not tracked through ownerReference either.

[popdaniel942]

In my opinion I think the best solution is to use argo's annotation+label tracking, it will know much better what resources it actually needs to track

[scholzj]

Ok ... but ... what exactly does that mean for someone without any Argo experience? Is some additional support needed for it? Or does it work today?

Strimzi already sets the app.kubernetes.io/instance label to the name of the Kafka CR. This should be set on all resources including the PVCs. However, these labels are very badly defined, so it is never really clear what should the value be and everyone seems expect some different value or way how it is propagated.

[AydinChavez]

From my personal point of view a mechanism (via argocd specific annotations - as described in comment https://github.com/orgs/strimzi/discussions/7205#discussioncomment-3704150) exists in order to tell argocd to not consider the strimzi pvcs for deletion and also to ignore those in the overall sync state. For my usecase there is no further todo necessary in strimzi for this topic (if I didn’t overlook something).

[popdaniel942]

It's something available in the latest versions of argo, I am using it, it already works. There is nothing to do on kafka side, it's just a configuration when deploying argo.It will add an argo annotation instead of label or both( with annotation+label) to all resources that need to be managed by argo. And this way argo will know better what resources to manage, it will know if some resources were created by someone else. And I already tried this with kafka and it won't see it's pvcs so they can't get out of sync.

[ilja-maslov]

OK, I had some time today to play with the tracking-id Argo annotation.

Here is somewhat outdated key discussion on how the references work and what exactly happens when they are not self-referenced.

TLDR; tracking-id value is expected to be either my-argo-app-name:my-api/my-kind:my-namespace/my-name or my-argo-app-name:other-api/other-kind:other-namespace/other-name, i.e. the first part refers to the Argo application name, the other 2 parts form the resource reference, which could be either self-referenced or could reference another object. When the resource under consideration is annotated referencing another resource, it will be represented in Argo UI at the top level, irrespective where in the hierarchy the referenced resource is. So, if we reference the Argo application or Kafka cluster or a StrimziPod set, provided this resource is considered to be managed by Argo, it will also result in the annotated resource to be tracked by Argo too, but will be always shown in the UI, something along these lines:

Not everything is completely clean, but it is a workable solution, here is what it looks like atm.

1. We have application.instanceLabelKey: argocd.argoproj.io/instance in the argocd-cm to avoid the conflict between the default label Argo tracks (app.kubernetes.io/instance) which could be updated by various external helm charts or operators and the one used by Argo to track the resources belonging to the app. This is a recommended Argo practice and the root cause why StrimziPod Set created PVCs don't seem to be associated with the Argo app
2. To enable annotation-based tracking in Argo CD, we need to add application.resourceTrackingMethod: annotation+label to the argocd-cm, this will switch the primary tracking to the annotations, yet still preserves the ownerReference that Argo uses implicitly to track the dependent resources
3. We annotate Argo app that manages the Strimzi cluster objects with a self-references tracking-id, because, by default, it would take the name from the parent App of apps pattern application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: dev-strimzi-flex:argoproj.io/Application:argocd/dev-strimzi-flex
  name: dev-strimzi-flex
  namespace: argocd
...

4. We add annotations that reference the app to the Kafka CR, we could also reference the Kafka or even corresponding StrimziPodSet object, but the UI will still show the lines going from your Argo app directly to the PVCs, so it seems that there is very little value at the time to use direct dependency reference to track the PVCs

Kafka.spec.kafka.template:
      persistentVolumeClaim:
        metadata:
          annotations:
            argocd.argoproj.io/tracking-id: "dev-strimzi-flex:argoproj.io/Application:argocd/dev-strimzi-flex"
...
Kafka.spec.zookeeper.template:
      persistentVolumeClaim:
        metadata:
          annotations:
            argocd.argoproj.io/tracking-id: "dev-strimzi-flex:argoproj.io/Application:argocd/dev-strimzi-flex"

5. Argo app is green and synced, no orphan pruning is happening, the only gotcha is that when I click on one of the PVCs in the Argo UI, the detailed representation is not ideal (Unknown status and it shows the Diff between the desired and actual states, I'd really prefer Argo to show no diff and no STATUS, but it is a minor issue

This is the best configuration I could have come up with so far.

One last thing, we run Argo 2.4.12, the argo issue link above suggests that the feature may be only working properly in 2.4.9+, but I haven't verified it.

[scholzj]

we could also reference the Kafka or even corresponding StrimziPodSet object

How would the annotation look like in this case? I guess it would it be something like this for example:

argocd.argoproj.io/tracking-id: "my-argo-app-name:core.strimzi.io/StrimziPodSet:my-namespace/my-cluster-kafka"

So it will still have the my-argo-app-name part, right? I wondered if there is some way for us to set this automatically, but we don't know the Argo Application name, so it would not be possible I guess.

[ilja-maslov]

we could also reference the Kafka or even corresponding StrimziPodSet object

How would the annotation look like in this case? I guess it would it be something like this for example:

argocd.argoproj.io/tracking-id: "my-argo-app-name:core.strimzi.io/StrimziPodSet:my-namespace/my-cluster-kafka"

Yes, the last part refers to the api/kind referenced in the middle part, somy-cluster-kafka or my-cluster-zookeeper depending on the SPS you'd like to reference. There are 2 modes for the reference, either you self-reference or not, when you are not self-referencing, but referencing any resource in the cluster (preserving the first part of the tracking-id), the sync becomes green and sync status Unknown for the PVC (see previous post).

Now, I have tried to self-reference for just one of the PVCs (PersistentClaimStorageOverride doesn't allow to override the annotations per broker, anyway) and I do not think I like what I see (non-self-referenced PVCs look much better, I think they must match on some prefix, maybe?):


It gets back to the same representation I had before turning on the annotation tracking, weird, probably still some bugs there.

I have also added argocd.argoproj.io/sync-options: Prune=false to PVC annotations to be able to prune the other resources automatically.

I've got a feeling from that feature discussion link from the previous post that Argo team prefers to display the non-self referenced resources at the top level in the UI, i.e. direct arrow from the Argo app to the resource. Maybe, I'll still point it to the corresponding SPS so that when they change their mind, we won't need to re-point :)

Update: ended up with these annotations, similar for the zookeeper:

      persistentVolumeClaim:
        metadata:
          annotations:
            argocd.argoproj.io/sync-options: Prune=false
            argocd.argoproj.io/tracking-id: "dev-strimzi-flex:core.strimzi.io/StrimziPodSet:dev-kafka/dev-cluster-kafka"

[AydinChavez]

What about following approach (the one I use):

Setting in the Argo config (as you do):

application.instanceLabelKey: argocd.argoproj.io/instance

Along with setting both following annotations in the PVC template from Strimzi:

argocd.argoproj.io/compare-options: IgnoreExtraneous
argocd.argoproj.io/sync-options: Prune=False

It should help to get rid of the faulty sync/diff results in ArgoCD.

[ilja-maslov]

The way I understood it (haven't tried), IgnoreExtraneous will just paint the Argo app green instead of yellow, but PVCs will stay out of sync. I believe that our ideal state is that we are not ignoring the out of sync errors, but rather teaching Argo to track these PVCs correctly as dependencies. Doesn't seem like there is a clean way to fully achieve this yet, unless one sets deleteClaim: true but this is highly undesirable for any production deployment.

[dbaarorapr]

@ilja-ag do you have a answer to above issues,we have same problems now that we need to remove the feature flag that allows us to use stateful sets as we need to upgrade to 0.36.1 strimzi and argo shows the out of sync (requires pruning) for PVCS and CRB?

[domenicbove]

Looks like Argo recommends this:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka
spec:
  kafka:
    template:
      persistentVolumeClaim:
        metadata:
          annotations:
            argocd.argoproj.io/compare-options: IgnoreExtraneous
            argocd.argoproj.io/sync-options: Prune=false
  zookeeper:
    template:
      persistentVolumeClaim:
        metadata:
          annotations:
            argocd.argoproj.io/compare-options: IgnoreExtraneous
            argocd.argoproj.io/sync-options: Prune=false

In the Argo UI it shows the Application is Synced, even though the PVCs are not synced. I think that's why also want you to set prune false to be safe!

[VijayPatil872]

Hi @scholzj,
We have deployed kafka with kRaft, with strimzi operator version 0.45.0. We are facing similar kind of issue related to PVCs. The ArgoCD is out of sync and PVCs are going in to progressing giving status pending deletion. All the kafka broker and controllers PVC appear as out of sync in ArgoCD, as if they need to get deleted. The PVCs get created but ArgoCD says they need to be deleted. We tried some workaround as stated above also we tried some workaround related to ignoreDifferences in ArgoCD application manifests but still issue persists. Is there any workable solution for this issue we can have??

[scholzj]

I have no idea. I do not know a thing about Argo and I do not use it.