Ideas to make this tool gitops-compliant

[yogeek]

Hi !
First, this tool is a nice idea 👍
However, in a gitops environment where a controller (argocd, flux...) reconciliates the actual state (cluster resources) with the desired state in GIT, I do not see how this can be used as is. Indeed, the pod image would be mutated by image-swapper and right after that, the gitops controller would change the image name back to the one defined in GIT...

Do you have any idea how this tool could work nicely in a GitOps environment ?

(of course, we could configure the GitOps controller to "ignore" the changes on the "image" field but it would not be a satisfying solution as we would lost the update feature...)

[estahn]

Hi @yogeek,

Interesting question. Could you share more about your environment and how you apply manifests, e.g. do you use helm for example?

Is the "GitOps controller" part of ArgoCD nowadays?

I'm not sure what the current state is with "GitOps", but we used ArgoCD heavily at the company where this tool was developed. CI would push the ArgoCD manifest to the cluster and the cluster would take care of reaching the desired state. We used helm charts for nearly all of this. In this instance, the manifests in Git would not match the state in Kubernetes as there was no point for that. You could restore the cluster with this, just not by checking out and applying the manifests. You would have to run the CI step applying the ArgoCD manifests, which in turn would install the helm charts. If you want to backup the cluster state there are better tools than Git.

Enrico

[yogeek]

Hi @estahn

We have ArgoCD installed in each of our clusters.
In each repo, a (jenkins) job generates YAML (with helm, kustomize or any other tool...) and push resulting k8s manifests (plain YAML) to the "state" branch of the same repo.
For each repo, an ArgoCD Application is created (with ArgoCD ApplicationSet Controller) and syncs the "state" branch
So, on our case, ArgoCD only does "kubectl apply" on plain YAML.

So if the "image" field is mutated, ArgoCd wil detect a drift and fix it by applying the state from GIT, and that would override the mutation by the original image.

Guillaume

[estahn]

In this case wouldn't you do the things k8s-image-swapper does within the Jenkins job? A couple of bash-lines with jq, docker push/pull to the target repo should suffice 🤔

What happens with things that introduce sidecars? I guess any of those tools wouldn't work either with the approach described.

[yogeek]

Maybe I have not made myself clear, sorry about that.

In fact, we already have a fully working CI/CD workflow and indeed we use a Jenkins shared lib that allows project to retrieve the ECR corresponding to the environment they want to deploy to and then we add the ECR prefix to the actual images (the images having been previously pushed in ECR by another mirror mechanism)

So today, our solution is fully gitops and present no major issue.

What I was wondering is if it was possible to avoid the whole custom part (sharedlib to replace images + script to mirror images to ECR) and make this step more transparent by using a solution designed to ease this kind of stuff.
Image swapper seems very interesting because it moves this logic in the cluster and makes it transparent to project but after reading the details, I understood that it would be impossible to use in parallel of our ArgoCD usage.

The goal of this discussion was to know if someone managed to use it in a gitops context, because in general, mutating webhooks are not compatible with such a pattern.

[estahn]

It should be possible to add a "CLI-mode" which takes the manifest and converts it as you described. If you already have a working solution moving it to k8s-image-swapper may not be a worthwhile undertaking.

I'm interested to see if other people have similar issues.

[jo-schulz]

Even though nothing has happened here for a long time, I would like to get in touch with a similar problem. Here, too, there is a problem with the GitOps controller and the different variants between Git and the “real” status in K8s. In our case, the GitOps Controller would find a difference to the Git version and write a remapping to the image as in Git. At this point I also have to say that I'm not sure if we would end up with an infinite loop of k8s-image-swapper (change to ECR repo) and the GitOps controller (change to image as in Git). An option to only push images to ECR and not to swap them in the cluster would be optimal. This way the project can already be implemented to store / save the images and in a later step it can be evaluated how the process with image swapping / remapping should be implemented.