use kubernetes-client informer, but pod finished event delayed by 3 hours.

[bestbugwriter]

My kubernetes-client version

            <groupId>io.fabric8</groupId>
            <artifactId>kubernetes-client</artifactId>
            <version>6.4.1</version>

My Kubernetes server version

serverVersion:
  buildDate: "2023-03-01T03:20:55Z"
  compiler: gc
  gitCommit: d0fc63469d8b77701822b1cbca6ea981a87a0c5c
  gitTreeState: clean
  gitVersion: v1.20.11-aliyun.1
  goVersion: go1.15.15
  major: "1"
  minor: 20+
  platform: linux/amd64

pod submit time 18:06, end time 18:08, but informer Callback state Finished at 21:16 and 22:14, so why it delay for such a long time？ Are there any other debugging methods that we need to operate.

[bestbugwriter]

@shawkins can you help me?

[shawkins]

so why it delay for such a long time？

I think you are saying that a job, or something else triggering a temporary pod, was started at 18:06 and was finished at 18:08. Then you saw an event from that pod at 21:16 - it's hard to say without knowing the full details of what is going on whether or why such a delay would occur.

If you had the pod status or metadata you could see what some of timestamps are, which could give a quick indication of whether kube had been recently operating on it. As for debugging you can see having io.fabric8.kubernetes.client.informers.impl at a debug level will give you a more complete picture of what is going on internally.

What kind of event generates that log you are highlighting - an update, delete? If it could be a delete - is the finalStateUknown flag set? That would mean the informer detected the deletion after fully relisting, rather than receiving a delete event from the watch. The debug log would definitely confirm if the informer temporarily lost connectivity to the api server.

Finally you'd have to ensure there's no possibility within your application logic for your event handlers to get stuck.