[FLINK-37515] Basic support for Blue/Green deployments

[schongloo]

What is the purpose of the change

This pull request adds basic support for Blue/Green deployments as outlined by FLIP-503 (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=337677648).

Brief change log

• A new FlinkBlueGreenDeploymentController controller, with the capability of managing the lifecycle of these deployments.
• A new corresponding CRD is introduced
• The new FlinkBlueGreenDeploymentController will manage this CRD and hide from the user the details of the actual Blue/Green (Active/StandBy) jobs.
• Delegate the lifecycle of the actual Jobs to the existing FlinkDeployment controller.

Verifying this change

• Test to verify a single basic deployment
• Test to verify a proper transition Blue -> Green
• Test to verify correct behavior when encountering an error before a transition
• Test to verify correct behavior when encountering an error during a transition

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changes to the CustomResourceDescriptors: no
• Core observer or reconciler logic that is regularly executed: no

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? docs / JavaDocs. UML diagrams for all scenarios added under docs/content/docs

Overview Diagram

[davidradl]

I am concerned about the number of TODOs. Things like dealing with errors and timing conditions should be in place in the initial drop of code - unless we can mark this capability as beta or the like. WDYT @gyfora ?

[schongloo]

@davidradl thanks for your review/comments. I've cleaned up all the outdated/invalid TODOs which pointed me 1 missing unit test, added. At this point the few remaining TODOs do not interfere with the main functionality and are not critical/open for discussion.

[hansh0801]

👍

[hansh0801]

when this feature gonna be released?

[schongloo]

Hi @hansh0801, trying to release it ASAP but I also want to make sure the logic and contracts are as robust as possible to minimize changes later. Looks pretty stable in my current testing and I've reopened it for review. Thanks!

[hansh0801]

cool ! and I'm also waiting for blue green phase 2 :)
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=337677650

[schongloo]

@hansh0801 indeed... Phase 2 will take a little bit longer due to its complexity but already working on it.

[maheepm-lyft]

Hi @schongloo @davidradl @gyfora , I hope you're doing well. I've been following this PR as blue-green deployments would solve a critical use case for our production needs. I noticed there have been no updates to this PR for a few weeks. Was wondering if there is anything blocking the release of this and if there is something myself or the OSS community could help with (testing, fixing any changes, etc). Thanks for all the work so far! 🙏

[gyfora]

Hi @schongloo @davidradl @gyfora , I hope you're doing well. I've been following this PR as blue-green deployments would solve a critical use case for our production needs. I noticed there have been no updates to this PR for a few weeks. Was wondering if there is anything blocking the release of this and if there is something myself or the OSS community could help with (testing, fixing any changes, etc). Thanks for all the work so far! 🙏

@maheepm-lyft , @schongloo is currently on holiday but once he is back we can try to merge this as soon as reasonably possible :)

In the meantime you can help us by trying out the current version and by providing a code review for the PR, that can definitely speed things up.

[gyfora]

@schongloo can you please re-open this PR against the main branch?

[maheepm-lyft]

Hi @schongloo @davidradl @gyfora , I hope you're doing well. I've been following this PR as blue-green deployments would solve a critical use case for our production needs. I noticed there have been no updates to this PR for a few weeks. Was wondering if there is anything blocking the release of this and if there is something myself or the OSS community could help with (testing, fixing any changes, etc). Thanks for all the work so far! 🙏

@maheepm-lyft , @schongloo is currently on holiday but once he is back we can try to merge this as soon as reasonably possible :)

In the meantime you can help us by trying out the current version and by providing a code review for the PR, that can definitely speed things up.

Great thanks! Taking us a bit of time to test this out in our stack, but we're working on it and I'll post any comments here.

[davidradl]

I suggest including some advice as to what to set this to in different circumstances. Same for deployment-deletion.delay

[davidradl]

nit maybe use a html list for this list

[schongloo]

@davidradl thank you for all your feedback. The past few weeks I was focused on mostly the high level refactoring, I'll address your comments ASAP.

[vikasap]

This will do replacement of the image tag as well. We would probably need to exclude that.

[maheepm-lyft]

+1 to what vikas said. This will change the name in places that are problematic. In my testing, I am facing problems so far in expected configmap name, savepoint path, and metrics reporters.

[maheepm-lyft]

Hi @schongloo , sorry to leave a comment after the PR is merged. I have been doing some testing with BlueGreen deployments, and I think this is replacing the name in several places where it should not be. For example, if I am using an initialSavepointPath with savepoint path in S3, it is replacing the app name in the path with app name + "-blue" or "-green", and the app is unable to find the corresponding savepoint. E.g. app_name becomes app_name-blue. If the savepoint path already has -blue in it (e.g. app_name-blue, then it is updating it to app_name-blue-blue. I am seeing similar issues related to a few other things like configmap names, although I haven't followed that code path all the way.

EDIT: added a comment elsewhere after looking into the bug a little more.