Tracing requests is unusable due to interaction with hyper_utils client

[ashtuchkin]

Current and expected behavior

Although I haven't seen it in the documentation, I was pleasantly surprised to find that kube-client crate adds a TraceLayer to it's stack by default, which makes it possible to publish spans for all requests to an opentelemetry setup.

Unfortunately we noticed an issue with it that prevents it from being usable - requests are marked as finished only after their underlying HTTP connections are disposed from the pool, which usually happens after 90s. Our usual requests are only ~20ms long, so it breaks all our duration graphs:

I've checked that the requests are returning the data correctly and quickly, it's just the tracing spans that are misleading.

Expected behavior is that the spans are correctly exited at the end of the request and not waiting for 90s.

Possible solution

After a couple hours of debugging, I identified an issue where hyper-util legacy client stores another reference to the span in the "dispatcher task" that is connected to the connection. That dispatcher task is kept alive when connection is put into the idle list, so that explains the behavior. I created a bug report here: hyperium/hyper#3904

One solution we can do at the kube-rs side is to stop requesting tracing feature from the hyper-utils crate. Not sure whether it's feasible for you.

Another solution would be to provide a way to disable the connection pooling. I know I can construct the whole client stack by reimplementing layers, but that seems like a tricky and brittle situation. It would be hard to do something like make_generic_builder myself because all the underlying methods like config.rustls_https_connector_with_connector, config.auth_layer, config.extra_headers_layer are all private.

So either a separate way to adjust hyper_util::client::legacy::Builder, or adding these methods to the public API would resolve this.

Additional context

No response

Environment

Linux 6.10.14-linuxkit #1 SMP Tue Apr 15 16:00:54 UTC 2025 aarch64 GNU/Linux

Client Version: v1.29.15
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.31.7-eks-4096722

Configuration and features

kube = { version = "1.0", features = ["runtime"] }
k8s-openapi = { version = "*", features = ["v1_31"] }

Affected crates

kube-client

Would you like to work on fixing this bug?

maybe

[ashtuchkin]

For context, here's the PR that adds this functionality (it's pretty recent): #1734
So the issue started with version 1.0, it wasn't there in 0.99.

[clux]

Thanks for raising and cross linking this. If this is legit, then it's probably something for upstream (tbh I barely understand the description in the linked pr so have stayed away).

---

That said, I am curious why you are seeing this like this and I'm not seeing it on tempo, but maybe it's because it's a simpler trace viewer. I get durations that look sensible:

(via instrumented fn on this stack)

This is probably unrelated, but be aware that you should probably not try to instrument an infinite watch via watcher as it tries to create main/application length spans (see also #741 (comment) and https://kube.rs/controllers/observability/#instrumenting). Thought I'd post just in case it's in any way helpful.