[SURE-10589] Fleet does not delete the "cluster-fleet-local-<clusterName>-<hash>" namespace when a cluster is deleted from Rancher UI.

[kkaempf]

SURE-10589

Issue description:

Deleting a cluster from Rancher UI does not clean up the cluster-fleet-local-<clusterName>-<hash> namespace from the SYSTEM project which is created by fleet. 

Business impact:

There is no business impact as such but customer has to delete the namespace manually. 

Troubleshooting steps:

We can clean up this namespace manually or restart the fleet-controller which would clean up the namespace.

Repro steps:

Create a Rancher provisioned downstream cluster.
This would create a namespace in the SYSTEM project for this cluster in cluster-fleet-local-<clusterName>-<hash> format. 
Delete the cluster from Rancher UI. 
This should ideally trigger fleet to remove this namespace once the cluster is deleted. However, the namespace remains as an orphan resource. 

Workaround:

Is a workaround available and implemented? yes
What is the workaround: Restart the fleet-controller to resolve the issue. 

Actual behavior:

Upon cluster deletion the cluster-fleet-local-<clusterName>-<hash> namespace in SYSTEM project is not cleaned up automatically. 

Expected behavior:

Once the cluster is deleted, all of its associated resources should be cleaned up from the cluster. 

Files, logs, traces:

Logs can be uploaded to the Jira on demand since this is easily reproducible.

[dharmit]

@kkaempf can I work on this one? I haven't worked on anything related to Fleet yet, so would like to get my hands dirty if that's OK.

[kkaempf]

It's yours, good luck 😉

[kkaempf]

@dharmit please post your updates here (or file a pull request, referencing this issue 😉 )

[dharmit]

I'm still looking into this trying to understand the flow of things. One thing I have been able to deduce so far is that namespace doesn't get deleted for older version too - I tried 2.10.8.

Beyond that, I'm looking for what could be the right place to make this change because a clusters.fleet.cattle.io resource seems to be created/owned by a clusters.provisioning.cattle.io. My mind doesn't have a full picture of code/architecture of fleet, so I'll keep posting my thoughts/learning here as things become clearer.

[dharmit]

Using the scripts setup-single-cluster in dev directory, I managed to create a development setup for fleet that doesn't involve/install Rancher. Then, I scaled down the fleet-controller deployment to zero replicas and ran the fleet-controller, fleet-agentmanagement and fleet-cleanup containers through IDE.

After inserting some breakpoints in the file internal/cmd/controller/agentmanagement/controllers/cluster/controller.go I expected to hit them when a downstream cluster is created using dev/setup-k3ds-downstream script, but that didn't happen. Maybe I'm expecting it wrong. Downstream cluster created this way doesn't seem to be creating a namespace in the dev setup either.

Maybe naive, but I added a clusters.OnRemove line after:

fleet/internal/cmd/controller/agentmanagement/controllers/cluster/controller.go

Line 54 in ed1d72b

clusters.OnChange(ctx, "managed-cluster-trigger", h.ensureNSDeleted)

but that didn't seem to be getting triggered upon removal of a downstream cluster.

If you have any pointers as to how to approach this one, let me know. :)
cc @manno @weyfonk @0xavi0 @kkaempf.