[Backport release-1.30] Wait for lease pool shutdown when stopping components #6019
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
twz123
force-pushed
the
backport-6017-to-release-1.30
branch
2 times, most recently
from
ed17715 to
d08611d
Compare
Signed-off-by: Tom Wieczorek <twieczorek@mirantis.com> (cherry picked from commit 6d9e36c)
Move server count and cluster config into the reconciliation goroutine and pass them as args to the write method. This will limit their scope and prevent them from being inadvertently updated by other goroutines. Remove the mutex, since the write method is only called by a single goroutine. Signed-off-by: Tom Wieczorek <twieczorek@mirantis.com> (cherry picked from commit f53d4b4)
twz123
force-pushed
the
backport-6017-to-release-1.30
branch
2 times, most recently
from
4ebfc59 to
2b89aeb
Compare
The controller lease counter requires the API server to be up and running in order to work properly. It cannot acquire or count leases before the API server is up, nor can it release the lease after the API server is shut down. Move the counter component down in the controller start method, so that it's started after the API server. Decouple the actual controller count value from the counter component by putting it directly into the start method. Let the counter component update that value, and let the konnectivity components consume those updates. Introduce internal/sync.Latest as a small abstraction for values that may change over time and will be consumed by multiple observers. Signed-off-by: Tom Wieczorek <twieczorek@mirantis.com> (cherry picked from commit 3a6b68b)
The context shouldn't be stored in structs, but passed as the first argument to functions and methods. Remove the context storage from the lease pool and pass it to the methods where it's actually needed. Signed-off-by: Tom Wieczorek <twieczorek@mirantis.com> (cherry picked from commit a7d8d12)
The components holding a lease pool weren't waiting for the pool to shutdown when they were stopped. They simply canceled an internal context. This led to a race condition with the API server shutdown. There was no guarantee that the API server was still up and running when the lease pool tried to invalidate its lease. This left the pool in a stale state within the cluster, causing leader transitions to take much more time than necessary. Add a new channel as a return value to LeasePool that callers can use to await pool termination. Wire this up in the components that use LeasePool. Signed-off-by: Tom Wieczorek <twieczorek@mirantis.com> (cherry picked from commit c37a021) (cherry picked from commit 6f23599) (cherry picked from commit 9098da3)
twz123
force-pushed
the
backport-6017-to-release-1.30
branch
from
2b89aeb to
1385991
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport to
release-1.30:See: