Delete obsolete bundles before writing new ones #4157
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
weyfonk
force-pushed
the
3770-bundle-path-update
branch
from
1f1d541 to
d47281f
Compare
weyfonk
force-pushed
the
3770-bundle-path-update
branch
from
d47281f to
aa139cc
Compare
weyfonk
force-pushed
the
3770-bundle-path-update
branch
from
0e2a429 to
33c74ef
Compare
When a GitRepo is updated such that a path to a bundle changes, but resources overlap between the old and new paths, there is a risk of obsolete bundles being cleaned up after newer bundles have been created. This may happen intermittently due to a race condition, and cause deletion of resources from the newer bundle, leading to the GitRepo status appearing as _Modified_. This commit adds end-to-end test cases exposing the issue, to make it easier to troubleshoot.
When a GitRepo is updated such that a path to a bundle changes, but resources overlap between the old and new paths, there was a risk of obsolete bundles being cleaned up after newer bundles had been created. This would happen intermittently due to a race condition, and cause deletion of resources from the newer bundle, leading to the GitRepo status appearing as _Modified_. To remedy this, `fleet apply` now deletes obsolete bundles synchronously before writing new ones, that last step happening asynchronously as it previously did.
This commit enables `fleet apply` to detect any possible resource overlaps, by kind and name, between: * obsolete bundles, which will be deleted because they no longer match any path in the GitRepo * new bundles coming from the GitRepo, to be created When such overlaps are detected, the bundle to be written has a newly populated `Overwrites` field, which is propagated to bundle deployments when targeting that bundle, specifying which resources (name, kind and namespace) this bundle deployment will overwrite from other bundles. This enables the agent to detect if a bundle deployment's `ModifiedStatus` features missing resources which may have been deleted when deleting a previous, obsolete bundle, for the bundle deployment to be redeployed. This last part does not quite work yet: * new deployments are triggered more than once * they do not result in updated, non-modified bundle deployment statuses. As a side note, it is unclear whether namespaces can even be reliably populated in bundle deployment overwrites, as a bundle resource's actual target namespace may depend on multiple factors.
In order to enable the agent to redeploy a bundle deployment with missing, overlapped resources, the corresponding Helm release must be deleted first, so that a new reconcile loop runs the deploy operation.
A couple of new comments should help our future selves, or our successors, understand why this new logic has been added.
weyfonk
force-pushed
the
3770-bundle-path-update
branch
from
33c74ef to
7e29c14
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a GitRepo is updated such that a path to a bundle changes, but resources overlap between the old and new paths, there was a risk of obsolete bundles being cleaned up after newer bundles had been created. This would happen intermittently due to a race condition, and cause deletion of resources from the newer bundle, leading to the GitRepo status appearing as Modified.
To remedy this,
fleet applynow deletes obsolete bundles synchronously before writing new ones, that last step happening asynchronously as it previously did.Refers to #3770.
Additional Information
Checklist
- [ ] I have updated the documentation via a pull request in thefleet-docs repository.
release/v0.13