Worker pod liveness probe unstable behavior

[ilya-vasiuk]

Describe the bug
The liveness probe in the lago-events-worker container fails when using the command with direct command substitution (grep $(hostname)). The probe times out even though the process is running correctly. This is due to the way the shell handles the command substitution in the current configuration.

To Reproduce
Steps to reproduce the behavior:

• Deploy the Lago application in a Kubernetes cluster.
• Observe that the liveness probe fails with a timeout, causing the container to restart even though the process is healthy.
• Verify by manually executing the command via kubectl exec to see that the command fails to find the hostname when using direct substitution.

$ kubectl exec -it lago-events-worker-xxxxxxx-yyyyy -n lago -- /bin/bash -l -c "bundle exec sidekiqmon processes | grep $(hostname) || exit 1"
Defaulted container "lago-events-worker" out of: lago-events-worker, wait-for-redis (init), wait-for-migrations (init)
command terminated with exit code 1

Expected behavior
The liveness probe should correctly verify that the process corresponding to the current hostname is running. When executed properly, the probe should succeed without timing out.

Support
Version: 1.17.4

Additional context
The issue appears to be related to the way command substitution is handled when passed directly as part of the liveness probe command. A working fix is to wrap the command in single quotes and assign the output of hostname to a variable within the shell:

livenessProbe:
  exec:
    command:
      - /bin/bash
      - -l
      - -c
      - 'host=$(hostname); bundle exec sidekiqmon processes | grep -q "$host" || exit 1'

This change prevents the premature interpolation of $(hostname) and allows the command to execute correctly within the container.

[jdenquin]

👋 @ilya-vasiuk, we added an other healthcheck for workers, its not documented right now.
You can do an http healthcheck on http://localhost instead of the sidekiqmon method

[ilya-vasiuk]

Thanks, @jdenquin, I'll try this approach

[emilejimini]

Hi @jdenquin,
Are there any plans to make the liveness probe configurable in the chart? From what I can tell, it's currently hardcoded and not adjustable. It would be really helpful to have this made configurable.
Thanks!

[till]

👋 @ilya-vasiuk, we added an other healthcheck for workers, its not documented right now. You can do an http healthcheck on http://localhost instead of the sidekiqmon method

Is this still appropriate? I also had to replace build in health checks, but port 80 does not seem to work.

[jdenquin]

Hello @emilejimini, yes we can make it customizable
@till yes its still appropriate

[till]

@jdenquin What is the port to use? I can try to send a PR to help if you'd like.

[till]

Found the port while poking around in the container, so here's my kustomize patch — seems to work with all workers.

- op: remove
  path: /spec/template/spec/containers/0/livenessProbe
- op: add
  path: /spec/template/spec/containers/0/livenessProbe
  value:
    httpGet:
      path: /
      port: 8080
      scheme: HTTP
    initialDelaySeconds: 5
    periodSeconds: 10

[jdenquin]

Y're totally right @till, port is 8080 for this liveness check : https://github.com/getlago/lago-api/blob/main/config/initializers/sidekiq.rb#L5

I'll fix it in the chart!

[till]

@jdenquin I am not sure if I my check is too intense. It seems to bounce the workers occationailly.

[jdenquin]

can you try with 30s ?

[till]

Yeah, tried, seems to be the same. Nothing in the logs, it just eventually times out and restarts the worker.

[jdenquin]

maybe OOM?

[jdenquin]

or is it because of the liveness probe?

[till]

I think it's the probe, it's "restarting failed container", and there's nothing happening yet.

[till]

It still restarts, or has restarted over 200 times.