How to test a controller which uses several clusters?

[guettli]

Up to now we use our custom way to reconcile in several clusters.

One paint-point we have:

How to test that my controller reconciles correctly in several clusters?

Is testing on your roadmap?

---

I am unsure if this is a good way to contact you. Please tell me if you prefer another channel for questions like this.

[everpeace]

I could say envtest is powerful enough to test multicluster controller in multipole clusters.

Although it hasn't been merged yet, you can see multi cluster controller test example here:

• providers/cluster-inventory-api/suite_test.go: create two clusters hub and member
• providers/cluster-inventory-api/provider_test.go: test for multicluster configmap controller

[embik]

Hey @guettli, perfectly fine to discuss this in an issue IMHO. You can also reach out on #sig-multicluster (Kubernetes Slack).

As @everpeace said, I think the way to go is to use envtest. I could see us adding some wrappers around that in this repository to help with spinning up a local and multiple target clusters (this part depends on what provider you plug into multicluster-runtime though; e.g. for the kcp provider, we have our own envtest that spins up kcp instances).

If there is interest, I think it makes sense to work on this together as part of this repository. @everpeace's contribution is a great starting point for this. Can you maybe expand a little bit on what your pain points with testing controllers are? And how are you hoping multicluster-runtime could help with them?

[guettli]

@embik My current pain point about testing controllers which connect to several clusters: There is no good documentation. Somehow I can solve that, and get something working. But good documentation would be great.

I guess for most cases it does not matter how these clusters get created. It would be great if multicluster-runtime could provide docs and tools to test with several clusters. I think using kind-clusters would be good, because only few people are experienced with kcp.