[Experimental] Support copying user resources to downstream clusters

[weyfonk]

A new experimental feature, gated behind environment variable EXPERIMENTAL_COPY_RESOURCES_DOWNSTREAM, enables Fleet to copy resources from the upstream cluster to all downstream clusters.

This introduces a new DownstreamResources field in bundle deployment options, which can be populated directly from a HelmOp's spec. Resources to be copied can be referenced through that field by name and kind, e.g.:

apiVersion: fleet.cattle.io/v1alpha1
kind: HelmOp
[...] # metadata
spec:
  helm:
    [...] # Helm options
  downstreamResources:
    - kind: Secret
      name: my-secret
    - kind: ConfigMap
      name: my-config

An important use case for this could be valuesFrom, which have requested secrets and/or config maps to be created directly on downstream clusters so far. This feature would eliminate that need.

Restrictions:

• Resources referenced in downstreamResources must live in the same namespace as the HelmOp resource.
• Only secrets and config maps are currently supported.

When referencing resources to be copied downstream, and the feature is enabled:

• the Fleet controller will copy those resources from the HelmOp's namespace to the (upstream) namespace of each targeted downstream cluster
• on each downstream cluster, the Fleet agent will:
  • copy those resources from the upstream namespace for the cluster to the deployment namespace, before the bundle deployment is deployed
  • delete these resources when the bundle deployment is deleted, unless keepResources is set to true on the bundle deployment.

Limitations:

• copying resources into the local cluster would not make much sense, and has not been tested
• this has only been tested with HelmOps, but it may work with GitOps too, since the implementation is independent of the bundle type
• Fleet does not monitor resources referenced by downstreamResources for changes. This could be part of a future iteration of this feature, but in the meantime, changes to secrets and config maps referenced for downstream copy will only be applied when a bundle is updated.

Refers to #3617.

Additional Information

Checklist

• I have updated the documentation via a pull request in the
  fleet-docs repository: Document propagation of resources to downstream clusters fleet-docs#362

[0xavi0]

Just minor things and my main concern would be the CreateOrUpdate call with a "does nothing" MutateFn.

[0xavi0]

yup, I also wonder if we should log instead...
We can keep it with error for now, the resource was created by the BundleDeployment so I guess it should behave like any other BundleDeployment

[0xavi0]

LGTM, thanks for the changes!