Skip to content

⚠️ [Warm Replicas] Implement warm replica support for controllers. #3192

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 39 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
8239300
[Warm Replicas] Implement warm replica support for controllers.
godwinpang Apr 9, 2025
73fc8fa
Remove irrelevant runnable_group.go code.
godwinpang Apr 14, 2025
be1b1c2
Rename ShouldWarmup.
godwinpang Apr 14, 2025
c9b99eb
fmt
godwinpang Apr 14, 2025
e7a2bbf
Change to atomic.Bool to avoid race in test.
godwinpang Apr 14, 2025
854987c
Address comments.
godwinpang Apr 29, 2025
072ad4b
Add ready check to block controller startup until warmup is complete.
godwinpang May 2, 2025
43118a3
Keep test helper structs private.
godwinpang May 2, 2025
b67bc65
Address comments.
godwinpang May 12, 2025
fc7c8c5
Fix lint.
godwinpang May 12, 2025
6bb4616
Address naming + comments from sbueringer.
godwinpang May 13, 2025
ccc7485
Refactor tests to use HaveValue.
godwinpang May 13, 2025
54f4fe3
Document + add UT for WaitForWarmupComplete behavior on ctx cancellat…
godwinpang May 14, 2025
667bb03
Add unit test that exercises controller warmup integration with manager.
godwinpang May 14, 2025
66e3be4
Add UT that verifies WaitForWarmupComplete blocking / non-blocking be…
godwinpang May 14, 2025
d9cc96b
Verify r.Others.startQueue in runnables test cases.
godwinpang May 14, 2025
65a04d5
Fix UT to verify runnable ordering.
godwinpang May 14, 2025
c201bfa
Fix UT for WaitForWarmupComplete blocking.
godwinpang May 15, 2025
5a13db4
Document !NeedLeaderElection+NeedWarmup behavior
godwinpang May 15, 2025
4879527
Fix test race.
godwinpang May 16, 2025
57acc77
Cleanup test wrapper runnables.
godwinpang May 16, 2025
1987b54
Make didStartEventSources run once with sync.Once + UT.
godwinpang May 16, 2025
a49f3a4
Rewrite Warmup to avoid polling.
godwinpang May 16, 2025
89f5479
Rename NeedWarmup to EnableWarmup.
godwinpang May 16, 2025
9d5ddfb
Clarify comment on Warmup.
godwinpang May 16, 2025
66f64f0
Move reset watches critical section inside of startEventSources.
godwinpang May 16, 2025
0563114
Add test to assert startEventSources blocking behavior.
godwinpang May 16, 2025
aa20ef5
Make Start threadsafe with Warmup + UT.
godwinpang May 16, 2025
c9a2973
Change warmup to use buffered error channel and add New method.
godwinpang May 19, 2025
79a7b95
Fail in warmup directly and rely on sync.Once for warmup thread-safet…
godwinpang May 20, 2025
c1d8ea4
Sync controller EnableWarmup comments.
godwinpang May 20, 2025
5df573f
Rename to startEventSourcesLocked and lock with c.mu
godwinpang May 21, 2025
d8650df
Address edge case for watch added after warmup completes.
godwinpang May 21, 2025
a03f404
Fix test description and set leaderelection==true
godwinpang May 21, 2025
dcf4b8b
Fix lint.
godwinpang May 21, 2025
ba51d28
Change shutdown order to shutdown warmup runnables in parallel with o…
godwinpang May 22, 2025
ea2aa0e
Fix test races by ensuring goroutines do not outlive their It blocks.
godwinpang May 22, 2025
730b30e
Block on source start on context cancel.
godwinpang May 22, 2025
bca3e2a
Guard access to c.Queue explicitly.
godwinpang May 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions pkg/config/controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,19 @@ type Controller struct {
// Defaults to true, which means the controller will use leader election.
NeedLeaderElection *bool

// EnableWarmup specifies whether the controller should start its sources when the manager is not
// the leader. This is useful for cases where sources take a long time to start, as it allows
// for the controller to warm up its caches even before it is elected as the leader. This
// improves leadership failover time, as the caches will be prepopulated before the controller
// transitions to be leader.
//
// Setting EnableWarmup to true and NeedLeaderElection to true means the controller will start its
// sources without waiting to become leader.
// Setting EnableWarmup to true and NeedLeaderElection is false is a no-op as controllers without
// leader election do not wait on leader election to start their sources.
// Defaults to false.
EnableWarmup *bool

// UsePriorityQueue configures the controllers queue to use the controller-runtime provided
// priority queue.
//
Expand Down
19 changes: 17 additions & 2 deletions pkg/controller/controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,16 @@ type TypedOptions[request comparable] struct {
//
// Note: This flag is disabled by default until a future version. It's currently in beta.
UsePriorityQueue *bool

// EnableWarmup specifies whether the controller should start its sources when the manager is not
// the leader. This is useful for cases where sources take a long time to start, as it allows
// for the controller to warm up its caches even before it is elected as the leader. This
// improves leadership failover time, as the caches will be prepopulated before the controller
// transitions to be leader.
//
// When set to true, the controller will start its sources without transitioning to be leader.
// Defaults to false.
EnableWarmup *bool
}

// DefaultFromConfig defaults the config from a config.Controller
Expand Down Expand Up @@ -124,6 +134,10 @@ func (options *TypedOptions[request]) DefaultFromConfig(config config.Controller
if options.NeedLeaderElection == nil {
options.NeedLeaderElection = config.NeedLeaderElection
}

if options.EnableWarmup == nil {
options.EnableWarmup = config.EnableWarmup
}
}

// Controller implements an API. A Controller manages a work queue fed reconcile.Requests
Expand Down Expand Up @@ -243,7 +257,7 @@ func NewTypedUnmanaged[request comparable](name string, options TypedOptions[req
}

// Create controller with dependencies set
return &controller.Controller[request]{
return controller.New[request](controller.ControllerOptions[request]{
Do: options.Reconciler,
RateLimiter: options.RateLimiter,
NewQueue: options.NewQueue,
Expand All @@ -253,7 +267,8 @@ func NewTypedUnmanaged[request comparable](name string, options TypedOptions[req
LogConstructor: options.LogConstructor,
RecoverPanic: options.RecoverPanic,
LeaderElected: options.NeedLeaderElection,
}, nil
EnableWarmup: options.EnableWarmup,
}), nil
}

// ReconcileIDFromContext gets the reconcileID from the current context.
Expand Down
66 changes: 66 additions & 0 deletions pkg/controller/controller_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -474,5 +474,71 @@ var _ = Describe("controller.Controller", func() {
_, ok = q.(priorityqueue.PriorityQueue[reconcile.Request])
Expect(ok).To(BeFalse())
})

It("should set EnableWarmup correctly", func() {
m, err := manager.New(cfg, manager.Options{})
Expect(err).NotTo(HaveOccurred())

// Test with EnableWarmup set to true
ctrlWithWarmup, err := controller.New("warmup-enabled-ctrl", m, controller.Options{
Reconciler: reconcile.Func(nil),
EnableWarmup: ptr.To(true),
})
Expect(err).NotTo(HaveOccurred())

internalCtrlWithWarmup, ok := ctrlWithWarmup.(*internalcontroller.Controller[reconcile.Request])
Expect(ok).To(BeTrue())
Expect(internalCtrlWithWarmup.EnableWarmup).To(HaveValue(BeTrue()))

// Test with EnableWarmup set to false
ctrlWithoutWarmup, err := controller.New("warmup-disabled-ctrl", m, controller.Options{
Reconciler: reconcile.Func(nil),
EnableWarmup: ptr.To(false),
})
Expect(err).NotTo(HaveOccurred())

internalCtrlWithoutWarmup, ok := ctrlWithoutWarmup.(*internalcontroller.Controller[reconcile.Request])
Expect(ok).To(BeTrue())
Expect(internalCtrlWithoutWarmup.EnableWarmup).To(HaveValue(BeFalse()))

// Test with EnableWarmup not set (should default to nil)
ctrlWithDefaultWarmup, err := controller.New("warmup-default-ctrl", m, controller.Options{
Reconciler: reconcile.Func(nil),
})
Expect(err).NotTo(HaveOccurred())

internalCtrlWithDefaultWarmup, ok := ctrlWithDefaultWarmup.(*internalcontroller.Controller[reconcile.Request])
Expect(ok).To(BeTrue())
Expect(internalCtrlWithDefaultWarmup.EnableWarmup).To(BeNil())
})

It("should inherit EnableWarmup from manager config", func() {
// Test with manager default setting EnableWarmup to true
managerWithWarmup, err := manager.New(cfg, manager.Options{
Controller: config.Controller{
EnableWarmup: ptr.To(true),
},
})
Expect(err).NotTo(HaveOccurred())
ctrlInheritingWarmup, err := controller.New("inherit-warmup-enabled", managerWithWarmup, controller.Options{
Reconciler: reconcile.Func(nil),
})
Expect(err).NotTo(HaveOccurred())

internalCtrlInheritingWarmup, ok := ctrlInheritingWarmup.(*internalcontroller.Controller[reconcile.Request])
Expect(ok).To(BeTrue())
Expect(internalCtrlInheritingWarmup.EnableWarmup).To(HaveValue(BeTrue()))

// Test that explicit controller setting overrides manager setting
ctrlOverridingWarmup, err := controller.New("override-warmup-disabled", managerWithWarmup, controller.Options{
Reconciler: reconcile.Func(nil),
EnableWarmup: ptr.To(false),
})
Expect(err).NotTo(HaveOccurred())

internalCtrlOverridingWarmup, ok := ctrlOverridingWarmup.(*internalcontroller.Controller[reconcile.Request])
Expect(ok).To(BeTrue())
Expect(internalCtrlOverridingWarmup.EnableWarmup).To(HaveValue(BeFalse()))
})
})
})
Loading
Loading