BUG: DriftDetectionManager; Missing CRDs (resources summaries) in child cluster.

[Banh-Canh]

Hello !

Very similar to a previous issue of mine #1205

The drift detection manager deployment deployed in child cluster seems to crash because of this error:

I1003 17:17:31.450873       1 manager.go:148] "Creating manager now."                                                                                                                        
I1003 17:17:31.454125       1 main.go:302] "failed to initialize manager no matches for kind \"ResourceSummary\" in version \"lib.projectsveltos.io/v1beta1\"" logger="setup"                
I1003 17:17:32.454981       1 manager.go:148] "Creating manager now."                                                                                                                        
I1003 17:17:32.458044       1 main.go:302] "failed to initialize manager no matches for kind \"ResourceSummary\" in version \"lib.projectsveltos.io/v1beta1\"" logger="setup"                
I1003 17:17:33.459073       1 manager.go:148] "Creating manager now."                                                                                                                        
I1003 17:17:33.461995       1 main.go:302] "failed to initialize manager no matches for kind \"ResourceSummary\" in version \"lib.projectsveltos.io/v1beta1\"" logger="setup" 

I'm unsure about how to troubleshoot this more, here is my repo of clusterprofiles/events.
https://github.com/Bealvio/flux-mgmt/tree/main/gitops/apps/sveltos

I'm using it quite extensively so if you could help me help you get more informations i would certainly put more effort into it !

Using v1.1.1, I can't tell if it's been introduced in the last version.

I hope it helps a little.

[gianlucam76]

Hi @Banh-Canh you are running Sveltos in with agents in the managed cluster right?

Addon-controller itself is the one that deploys ResourceSummary and it does it before deploying the drift detection manager deployment.

Can you share the steps that brought to this condition? I assume this was not a fresh install

[Banh-Canh]

Hi @Banh-Canh you are running Sveltos in with agents in the managed cluster right?

Actually, it's in mode 2, with all agents centralized in management cluster

Addon-controller itself is the one that deploys ResourceSummary and it does it before deploying the drift detection manager deployment.

Can you share the steps that brought to this condition? I assume this was not a fresh install

Enabling ContinuousWithDriftDetection brought this, I wasn't using it for a while so I discovered this just now.

It is indeed not a fresh install.
Also, I have 2 child cluster actually, and only one of the drift detection manager is crashing and throwing error about missing CRDs. 😅

[gianlucam76]

Thanks. so drift-detection-manager deployments are running in your management cluster. The ResourceSummary instance there is created by the helm install.

But it appears that again the drift-detection-manager is checking for that resource in the managed cluster (and rightfully it wont be present there).

Can you share the YAML of one drift-detection-manager deployment that you currently have in your management cluster and that is crashing/reporting this error?

[gianlucam76]

and also the drift-detection-manager log please. thank you

[Banh-Canh]

Ok so there is something wrong happening, because my drift detection manager are running in managed cluster, not the management cluster. While all the agent are running in management cluster ..

Let me triple check all that, but i'm pretty certain I'm running in mode 2

[gianlucam76]

oh is it possible during upgrade accidentally you switched mode? That is not supported

[Banh-Canh]

oh is it possible during upgrade accidentally you switched mode? That is not supported

I'm pretty sure I did not
https://github.com/Bealvio/flux-mgmt/commits/main/gitops/apps/sveltos/helmrelease.yaml

Can't eliminate manual apply but I doubt it. whatever anyway aha it's just weird on my end again

Can you give me some insight on how I can fix this ? Not expecting big help since it's not supported but maybe some tips :)

[gianlucam76]

Verify you are running with agent in the management cluster. Check the addon-controller deployment. It must have this arg agent-in-mgmt-cluster=true

Then you can delete the drift-detection-manager running in your managed clusters.

Change the ClusterProfile SyncMode from ContinuousWithDriftDetection to Continuous.
Wait for all ClusterSummary to be marked as Provisioned.

Set log verbosity for addon controller to debug
kubectl edit debuggingconfigurations.lib.projectsveltos.io

Change the ClusterProfile SyncMode from Continuous to ContinuousWithDriftDetection.

the drift-detection-manager deployments will be deployed in the management cluster (and nothing in the managed cluster) and all should work.

If any issue happens, please collect addon-controller log/YAML and the drift-detection-manager log/YAML

[Banh-Canh]

actually I noticed i have the drift manager in both managed and management cluster

sveltos addon controller is correctly configured with the arg. I deleted the drift manager in child cluster. I'll see how it goes from now, but I already did this sometime earlier today.

I'll now check carefully as I'm now sure about what is the expected way the drift manager is supposed to be deployed.

I'll get back to you once I know more or to close the issue, thank you, you've been a big help. :)

[gianlucam76]

Thanks. There are two paths that deploy a drift-detection-manager. one is when addon-controller starts. Can you please restart addon-controller and see which one of those logs you see.

func upgradeDriftDetection(ctx context.Context, logger logr.Logger) {
	if getAgentInMgmtCluster() {
		logger.V(logs.LogInfo).Info("upgrade drift detection on management cluster")
		upgradeDriftDetectionDeploymentsInMgmtCluster(ctx, logger)
		return
	}

	logger.V(logs.LogInfo).Info("upgrade drift detection on managed clusters")
	upgradeDriftDetectionDeploymentsInManagedClusters(ctx, logger)
}

The second one is when you set for the first time SyncMode to ContinuousWithDriftDetection

[Banh-Canh]

│ I1004 08:36:53.387162       1 drift_detection_upgrade.go:40] "upgrade drift detection on management cluster" logger="clustersummaryreconciler"                                            │

I'll gather some more debugging data, but it seems each time i get a drift, it redeploys the drift manager on child cluster even though there is already an instance on the management cluster for each child cluster

[gianlucam76]

Thank you @Banh-Canh . This is helpful.

If you are deploying Kustomize resources with Sveltos (setting KustomizationRefs section) I found the bug. Pushed this PR

While HelmCharts and PolicyRefs works just fine with ContinuosWithDriftDetection.

When there is a drift, flow is same of when SyncMode is set to ContinuousWithDriftDetection.
The drift-detection pod (which at this point in your case is in the management cluster) detects the drift and reset the ClusterSummary causing addon-controller to redeploy it.

So you should see this log

func deployDriftDetectionManagerInManagementCluster(ctx context.Context, restConfig *rest.Config,
	clusterNamespace, clusterName, mode string, clusterType libsveltosv1beta1.ClusterType,
	patches []libsveltosv1beta1.Patch, logger logr.Logger) error {

	logger.V(logs.LogDebug).Info("deploy drift-detection-manager in management cluster")
	driftDetectionManagerYAML := string(driftdetection.GetDriftDetectionManagerInMgmtClusterYAML())

and not see this

// deployDriftDetectionManagerInManagedCluster deploys the drift-detection-manager component within the managed cluster.
func deployDriftDetectionManagerInManagedCluster(ctx context.Context,
	clusterNamespace, clusterName, applicant, featureID, mode string,
	clusterType libsveltosv1beta1.ClusterType, patches []libsveltosv1beta1.Patch, logger logr.Logger) error {

	// Sveltos resources are deployed using cluster-admin role.
	cacheMgr := clustercache.GetManager()
	remoteRestConfig, err := cacheMgr.GetKubernetesRestConfig(ctx, getManagementClusterClient(),
		clusterNamespace, clusterName, "", "", clusterType, logger)
	if err != nil {
		logger.V(logs.LogInfo).Error(err, "failed to get cluster rest config")
		return err
	}

	logger.V(logs.LogDebug).Info("deploy drift-detection-manager in managed cluster")
	driftDetectionManagerYAML := string(driftdetection.GetDriftDetectionManagerYAML())

[Banh-Canh]

Wow thank you, you found it so quickly with so little informations :) I'll test how it goes then.

Thank you so much.