helm akhq: liveness/readiness probe failed

[PDVJAM]

Hey!
We use an official helm chart, deployed per documentation, nothing unusual, nothing changed.
But we always see liveness probe errors:

Normal   Killing    4m30s (x2 over 5m30s)  kubelet            Container akhq failed liveness probe, will be restarted
  Warning  Unhealthy  4m29s (x9 over 5m49s)  kubelet            Readiness probe failed: Get "http://100.64.20.45:28081/health": dial tcp 100.64.20.45:28081: connect: connection refused
  Warning  Unhealthy  50s (x16 over 5m50s)   kubelet            Liveness probe failed: dial tcp 100.64.20.45:28081: connect: connection refused

pod log:

[eis@aws0v20prodbld01 ~]$ kubectl logs kafka-gui-akhq-65fcb6f9f8-qmc64
2021-11-25 11:57:56,532 INFO  main       i.m.c.e.DefaultEnvironment Established active environments: [k8s, cloud, ec2, secrets]
2021-11-25 11:57:58,684 INFO  main       i.m.runtime.Micronaut      Startup completed in 2253ms. Server Running: http://kafka-gui-akhq-65fcb6f9f8-qmc64:8080

What might be wrong with the probe? Tried on several k8s clusters, it's always the same. Except for disabling this probe - what can we do?

[tchiotludo]

Tell me more :

• the version of akhq ?
• the version of helm charts ?

[PDVJAM]

appVersion: 0.19.0
version: 0.2.6

Telnet is able to connect to the management port, but curl is unable to get any data from endpoints - the connection is just stuck.

I have found a topic with how to enable debug log, hoping it will help to find what is wrong, but it's unclear where exactly should I add that configuration block. I have tried to add it to helm values and to the app's configuration, but didn't notice any difference in logs.

[tchiotludo]

Really strange ...
Can you provide the full deployment.yaml deployed on your cluster ?

[PDVJAM]

deploy:

dpavlovskyi@dpavlovskyi:[~]: k -n nft-dev get deploy akhq -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: akhq
    meta.helm.sh/release-namespace: nft-dev
  creationTimestamp: "2021-11-26T07:20:50Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: akhq
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: akhq
    helm.sh/chart: akhq-0.2.6
  name: akhq
  namespace: nft-dev
  resourceVersion: "64755280"
  uid: 353d28de-8d03-4811-ba1f-2526715ef32d
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: akhq
      app.kubernetes.io/name: akhq
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/config: 37a61a201b3f10a10d91c2831ccc998885ac03c7a692b92e78b61298a47cbc3b
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: akhq
        app.kubernetes.io/name: akhq
    spec:
      containers:
      - image: tchiotludo/akhq:0.19.0
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: management
          timeoutSeconds: 1
        name: akhq
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 28081
          name: management
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: management
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /app/application.yml
          name: config
          subPath: application.yml
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: akhq
        name: config

events:

  Warning  Unhealthy  2m22s                kubelet            Readiness probe failed: Get "http://10.244.1.97:28081/health": dial tcp 10.244.1.97:28081: connect: connection refused
  Warning  Unhealthy  5s (x14 over 2m15s)  kubelet            Readiness probe failed: Get "http://10.244.1.97:28081/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

curl from under the container:

akhq@akhq-7844b59cd9-4bbnr:/app$ curl http://10.244.1.97:28081/health -v
*   Trying 10.244.1.97:28081...
* Connected to 10.244.1.97 (10.244.1.97) port 28081 (#0)
> GET /health HTTP/1.1
> Host: 10.244.1.97:28081
> User-Agent: curl/7.74.0
> Accept: */*
>
^C

akhq@akhq-7844b59cd9-4bbnr:/app$ curl -i -X POST -H "Content-Type: application/json" \
       -d '{ "configuredLevel": "TRACE" }' \
       http://localhost:28081/loggers/org.akhq
^C

log:

 __  __ _                                  _
|  \/  (_) ___ _ __ ___  _ __   __ _ _   _| |_
| |\/| | |/ __| '__/ _ \| '_ \ / _` | | | | __|
| |  | | | (__| | | (_) | | | | (_| | |_| | |_
|_|  |_|_|\___|_|  \___/|_| |_|\__,_|\__,_|\__|
  Micronaut (v2.5.5)

2021-11-25 11:35:23,943 INFO  main       i.m.c.e.DefaultEnvironment Established active environments: [k8s, cloud, ec2, secrets]
2021-11-25 11:35:26,016 INFO  main       i.m.runtime.Micronaut      Startup completed in 2173ms. Server Running: http://akhq-7844b59cd9-4bbnr:8080

[tchiotludo]

Nothing special here ...
try to add this on values.yaml :

akhq:
  configuration:
    logger:
      levels:
        root: DEBUG

It will start a very verbose log maybe to validate that the management port is the good one.

Can you also provide the full values.yaml ?

[PDVJAM]

values.yaml:

ingress:
  enabled: false
  istio: false
  useClusterGateway: true # If false namespace gateway will be used
  domain_prefix: ""
  domain: ""
  tls:
    enabled: true
    secretName: "tls-secret"
  annotations: {}
  hosts:
    - host: kafka-gui # FQDN template: {{ .host }}-{{ $namespace }}.{{ $domain }}
      paths:
        - path: "/"
          serviceName: "kafka-gui-akhq"
          servicePort: 80 # service port which will be exposed
akhq:
  configuration:
    logger:
      levels:
        root: DEBUG
  imagePullSecrets:
    - name: registry-secret
  image:
    repository: "000.dkr.ecr.us-west-2.amazonaws.com/xxx/tchiotludo/akhq"
    tag: latest
  secrets: |
    akhq:
      connections:
        app:
          properties:
            bootstrap.servers: "kafka:9092"

yes, now I see debug log. really huge one.

dpavlovskyi@pdv:[~]: grep 28081 kafka-gui.log
dpavlovskyi@pdv:[~]:

nothing related to management port. what should I look for?

[tchiotludo]

I see :

  image:
    repository: "000.dkr.ecr.us-west-2.amazonaws.com/xxx/tchiotludo/akhq"

Are you sure that you use a cloned image from last version ?
The management port have change from 8081 to 28081 in 0.18.0 as I remember

[PDVJAM]

Pretty sure:

Containers:
  akhq:
    Container ID:   containerd://c45341f84c486773f80b2976989d9fb9028a61e03b18ee5bc5175fb54b1471fe
    Image:          tchiotludo/akhq:0.19.0
    Image ID:       docker.io/tchiotludo/akhq@sha256:053d3b691143671f1fad0dcac005af04ef8939be56df6db28bfe96dc0a53d326
    Ports:          8080/TCP, 28081/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Fri, 26 Nov 2021 11:18:04 +0200
    Ready:          False
    Restart Count:  0
    Liveness:       tcp-socket :management delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:management/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /app/application.yml from config (rw,path="application.yml")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pdvgb (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      akhq
    Optional:  false
  kube-api-access-pdvgb:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  39s               default-scheduler  Successfully assigned nft-dev/akhq-68cb85d6f5-rm8qf to nft-worker-pool-uqncv
  Normal   Pulling    38s               kubelet            Pulling image "tchiotludo/akhq:0.19.0"
  Normal   Pulled     37s               kubelet            Successfully pulled image "tchiotludo/akhq:0.19.0" in 751.242196ms
  Normal   Created    37s               kubelet            Created container akhq
  Normal   Started    37s               kubelet            Started container akhq
  Warning  Unhealthy  36s               kubelet            Readiness probe failed: Get "http://10.244.1.91:28081/health": dial tcp 10.244.1.91:28081: connect: connection refused
  Warning  Unhealthy  8s (x3 over 28s)  kubelet            Readiness probe failed: Get "http://10.244.1.91:28081/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I have tried on several clusters.

[tchiotludo]

The only things I will do for now, is to connect inside the pod and to curl the directly the management port.
Also you can edit manually the deployment to remove the healthcheck and see If the ui is displayed well

[PDVJAM]

It would be nice if you check this from your side.

dpavlovskyi@EIS:[~/1]: k -n nft-dev exec -it akhq-7844b59cd9-ckmmp bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
akhq@akhq-7844b59cd9-ckmmp:/app$ curl http://10.244.1.37:28081/health
^C

akhq@akhq-7844b59cd9-ckmmp:/app$ curl -i -X POST -H "Content-Type: application/json" \
       -d '{ "configuredLevel": "TRACE" }' \
       http://localhost:28081/loggers/org.akhq
^C

Because for me curl doesn't work if I try to connect to 28081 port.

[tchiotludo]

root@44462ce86f82:/app# curl   http://localhost:28081/health
{"name":"akhq","status":"UP","details":{"compositeDiscoveryClient()":{"name":"akhq","status":"UP"},"service":{"name":"akhq","status":"UP"},"diskSpace":{"name":"akhq","status":"UP","details":{"total":108311339008,"free":4586536960,"threshold":10485760}}}}root@44462ce86f82:/app# 

works directly without any issue on many env.

• Do you have pod network policy to limit internet access ?
• Can you remove the healthcheck and see id the application works ?

[PDVJAM]

root@44462ce86f82:/app# curl   http://localhost:28081/health
{"name":"akhq","status":"UP","details":{"compositeDiscoveryClient()":{"name":"akhq","status":"UP"},"service":{"name":"akhq","status":"UP"},"diskSpace":{"name":"akhq","status":"UP","details":{"total":108311339008,"free":4586536960,"threshold":10485760}}}}root@44462ce86f82:/app# 

works directly without any issue on many env.

* Do you have pod network policy to limit internet access ?

* Can you remove the healthcheck and see id the application works ?

Hey!

1. No policies.
2. Yes, it works if I remove health checks from the deployment. Maybe it would be a nice idea to make liveness/readiness probes optional?