Skip to content

Conversation

@prune998
Copy link

This PR add a new Promotion Step that caan merge multiple YAML files into a single file.

This is useful in the scenario where, for example, multiple promotions (ex: one per cluster) merges content in the same repo/branch.
Then Helm will pick up the values from multiple locations, but updating global values could impact many clusters at the same time.

With this step, we are able to pre-merge global values with more specific values and only commit one single values.yaml file for each specific cluster. No more globals.

@prune998 prune998 requested review from a team as code owners July 17, 2025 00:06
@netlify
Copy link

netlify bot commented Jul 17, 2025

Deploy Preview for docs-kargo-io ready!

Name Link
🔨 Latest commit b537d59
🔍 Latest deploy log https://app.netlify.com/projects/docs-kargo-io/deploys/68f62a8ccbddae0008b57f3c
😎 Deploy Preview https://deploy-preview-4634.docs.kargo.io
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@prune998 prune998 force-pushed the prune/yaml-merger branch from 89bbf3d to f13147f Compare July 17, 2025 00:14
@prune998 prune998 changed the title added a YAML Merger PromotionStep feat: add a YAML Merger PromotionStep Jul 17, 2025
@prune998 prune998 force-pushed the prune/yaml-merger branch 3 times, most recently from bde39d7 to e3bed04 Compare July 17, 2025 15:12
@prune998
Copy link
Author

Any way to get someone trigger the CI build ?
thx

@codecov
Copy link

codecov bot commented Aug 6, 2025

Codecov Report

❌ Patch coverage is 65.38462% with 36 lines in your changes missing coverage. Please review.
✅ Project coverage is 55.47%. Comparing base (d6b2774) to head (b537d59).
⚠️ Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
pkg/promotion/runner/builtin/yaml_merger.go 62.02% 25 Missing and 5 partials ⚠️
pkg/yaml/yaml.go 76.00% 4 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4634      +/-   ##
==========================================
+ Coverage   55.44%   55.47%   +0.02%     
==========================================
  Files         404      405       +1     
  Lines       36426    36539     +113     
==========================================
+ Hits        20198    20270      +72     
- Misses      15265    15298      +33     
- Partials      963      971       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@fuskovic
Copy link
Member

fuskovic commented Aug 6, 2025

Stopped by for a drive-by review.

Concept looks reasonable to me.

Related to #4766

Linter needs to be satisfied.

I'll defer the rest to @hiddeco and @krancour

@prune998 prune998 force-pushed the prune/yaml-merger branch from 3268947 to e36bfe8 Compare August 6, 2025 14:19
@prune998
Copy link
Author

prune998 commented Aug 6, 2025

thanks for the review @fuskovic. Everything should be resolved + rebased now.
I'll keep watching the workflows when they run, in case any other issue is detected.

@krancour krancour added kind/enhancement An entirely new feature priority/low Low commitment from maintainers; progress is likely to be community-driven area/controller Affects the (main) controller needs discussion A maintainer explicitly requests no action be taken without further discussion kind/proposal Indicates maintainers have not yet committed to a feature request labels Aug 12, 2025
@prune998
Copy link
Author

Anyone to review this ?

@krancour
Copy link
Member

krancour commented Aug 16, 2025

@prune998 I know this isn't what anyone wants to hear, but our priorities are elsewhere right now and there may be a bit of a wait for a thorough review.

Some discussion may also be in order. I see this is meant to address #4766, but no maintainer triaged that feature request or moved it out of proposal status. It's a gamble putting the effort into a feature without those conversations happening first.

In all probability, this is a worthwhile feature and it will be reviewed eventually and probably merged, but some patience may be required.

If it makes you feel any better, I have PRs open to some other projects that are old enough to be starting kindergarten. Our backlog isn't that bad, thankfully.

@krancour
Copy link
Member

Small addendum... there are some notoriously well-known gotchas when merging YAML, so this isn't a review I'd rush through and that contributes to the above.

@prune998
Copy link
Author

Thanks for the comment @krancour . I understand you're busy with your customers features.
The way Kargo is designed forces us to embed the promotionStep functions inside the code, which creates friction. Got it.

What you could do, would be

  • have someone do a quick review, ensuring that it works and it is useful, and more importantly, is not a security or stability risk
  • mark that function as alpha, not supported
  • add me as support of the feature and assign me comments/issues if any
  • release the feature

This way, your team will not be much impacted, the features is released, and you have a "feature maintainer".

This is just a proposition and you can refuse it. I'll wait, or keep using my forked version... but it will be a miss :)
Thanks for your time and effort anyway.

Side note on YAML merging: yeah, there's a ton of gotchas... this is why I used a "well known" library to do it and not try to re-implement it myself. The library I used is also opinionated. But it is intended for K8s yaml style. I also tested others, with varying results... Anyway, they all have the same limitations (biggest one being no merge of lists, latest list takes precedence), like Helm does, so people knowing Helm will not be disturbed.
Once I'm sure you're interrested in this feature and the code, I can add more examples in the docs and test cases in the code to better demonstrate all sort of merges. I just don't want to spend too much time if you are to refuse it. All I can say is that it is working "like Helm" for me at the moment.

Signed-off-by: Prune <prune@lecentre.net>
Signed-off-by: Prune <prune@lecentre.net>
Signed-off-by: Prune <prune@lecentre.net>
Signed-off-by: Prune <prune@lecentre.net>
Signed-off-by: Prune <prune@lecentre.net>
Signed-off-by: Prune <prune@lecentre.net>
Signed-off-by: Prune <prune@lecentre.net>
Signed-off-by: Prune <prune@lecentre.net>
Signed-off-by: Prune <prune@lecentre.net>
Signed-off-by: Prune <prune@lecentre.net>
Signed-off-by: Prune <prune@lecentre.net>
Signed-off-by: Prune <prune@lecentre.net>
Signed-off-by: Prune <prune@lecentre.net>
Signed-off-by: Prune <prune@lecentre.net>
Signed-off-by: Prune <prune@lecentre.net>
Signed-off-by: Prune <prune@lecentre.net>
@prune998
Copy link
Author

All requested modifications applied.
Thanks for the full review @fykaa

@fykaa
Copy link
Contributor

fykaa commented Oct 17, 2025

thanks @prune998 I'll do a final cleanup before approving this PR

Signed-off-by: Faeka Ansari <faeka6@gmail.com>
@fykaa fykaa force-pushed the prune/yaml-merger branch from 596e77a to d87a0c4 Compare October 17, 2025 15:38
Copy link
Contributor

@fykaa fykaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reviewed this PR and it works correctly for both cases - with and without ignoreMissingFiles. in its current implementation, it will error when all files in inFiles are missing and there's nothing to merge.

Deferring to @krancour for the final merge.

@prune998
Copy link
Author

all set !! Many thanks @fykaa. Hope to see that merged soon ! Thanks all for you help

Signed-off-by: Faeka Ansari <faeka6@gmail.com>
@prune998
Copy link
Author

are we good to merge then ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/controller Affects the (main) controller kind/enhancement An entirely new feature kind/proposal Indicates maintainers have not yet committed to a feature request needs discussion A maintainer explicitly requests no action be taken without further discussion priority/low Low commitment from maintainers; progress is likely to be community-driven

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants