fix: race condition when stopping health check during shutdown

[cre8ivejp]

Fix #2178

I noticed that after stopping the health check, it returns 503, but after a few seconds, it starts returning 200 again, causing issues making the pod ready again to receive new requests.

Things done

• Fix race condition when stopping health check during shutdown
• Changed to fail readiness probe, right after getting the sigterm
• Added unit tests

[cre8ivejp]

The problem was the Run() running in another goroutine.
So, we ensure that it will skip the check after stopping the http server.

[Ubisoft-potato]

https://github.com/bucketeer-io/bucketeer/blob/chore-improve-shutdown/pkg/subscriber/cmd/server/server.go#L364-L370

We should add sleep propagationDelay in subscriber service, too.

// Wait for K8s endpoint propagation
// This prevents "context deadline exceeded" errors during high traffic.
time.Sleep(propagationDelay)

[cre8ivejp]

https://github.com/bucketeer-io/bucketeer/blob/chore-improve-shutdown/pkg/subscriber/cmd/server/server.go#L364-L370

We should add sleep propagationDelay in subscriber service, too.

// Wait for K8s endpoint propagation
// This prevents "context deadline exceeded" errors during high traffic.
time.Sleep(propagationDelay)

@Ubisoft-potato, the reason I didn't use a delay is because the subscriber doesn't receive http or grpc requests. It only pulls messages from pubsub.

[Ubisoft-potato]

https://github.com/bucketeer-io/bucketeer/blob/chore-improve-shutdown/pkg/subscriber/cmd/server/server.go#L364-L370
We should add sleep propagationDelay in subscriber service, too.

// Wait for K8s endpoint propagation
// This prevents "context deadline exceeded" errors during high traffic.
time.Sleep(propagationDelay)

@Ubisoft-potato, the reason I didn't use a delay is because the subscriber doesn't receive http or grpc requests. It only pulls messages from pubsub.

Understand! Then we also dont need wait propagationDelay in the deployment.yaml
Thanks for explaination!

[Ubisoft-potato]

LGTM! 👍