[SURE-10335] GitRepo Bundle shows hash mismatch error after upgrading rancher to 2.10.6 from 2.10.4

[skanakal]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

After upgrading rancher from v2.10.4 to v2.10.6, one of the bundle "'k8s005-uat2-deployments-bi-reports"of a GitRepo shows error:

Content does not match hash. Got x, expected y
And its corresponding deployment is not deployed in the cluster.

Expected Behavior

The bundle should continue to work even after Rancher upgrade.

Steps To Reproduce

No response

Environment

- Architecture: 
- Fleet Version: 0.11.8
- Cluster:
  - Provider: RKE2
  - Options:
  - Kubernetes Version: 1.31

Logs

msg":"Reconciler error","controller":"bundle","controllerGroup":"fleet.cattle.io","controllerKind":"Bundle","Bundle":{"name":"deployments-xxx-xxx","namespace":"fleet-default"},"namespace":"fleet-default","name":"deployments-xxx-xxx","reconcileID":"cfcf9eeb-38e1-4192-b82f-2ba30bce1b3a","error":"Operation cannot be fulfilled on bundles.fleet.cattle.io \"deployments-xxx-xxx\": StorageError: invalid object, Code: 4, Key: /registry/fleet.cattle.io/bundles/fleet-default/deployments-xxx-xxx, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: xxxxxxxxxx, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:224"}

Anything else?

• Tried to delete this affected bundle and force update the Gitrepo, the bundle gets re-created but that didn't help
• Tried to restart fleet-controller and gitjob pods no success
• Tried to update this "deployments-xxx-xxx" bundle's fleet.yaml with a test text to see if fleet from rancher side triggers new hash key, but the error was same.
• Renamed the bundle to see if that helps...
• Removed finalizer deployments-xxx-xxx from content resource and Deleted content resource, then force update the gitrepo but that didn't help.

[manno]

I run into the storage error Operation cannot be fulfilled on bundles.fleet.cattle.io \"deployments-xxx-xxx\": StorageError: invalid object, Code: 4, Key: /registry/fleet.cattle.io/bundles/fleet-default/deployments-xxx-xxx, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: xxxxxxxxxx, UID in object meta: " when removing a bundle with many bundledeployments. Afterwards I have to restart the fleet-controller, so that processing can continue.

Tehere are some related upstream issues and this might be fixed in k8s 1.34

• [bug] StorageError: invalid object kubernetes-sigs/controller-runtime#1881
• [bug] StorageError: invalid object kubernetes/kubernetes#124347

[weyfonk]

Closing as not planned, as this requires more information on reproducing the issue; we may reopen it if/when more information is available.