Half-open HBONE connections cause 503 errors when Pod IPs are reused in Ambient Mode ?

[Dylan-KW]

Problem Summary

In Istio Ambient Mode with waypoint proxy enabled, we're experiencing intermittent 503 Upstream connection termination errors during pod-to-pod communication (both are in mesh). The issue occurs when Pod IP addresses are reused after the AWS VPC CNI's 30-second cooldown period, causing RST packets due to stale half-open HBONE connections held by the waypoint proxy.

Environment

Istio Version: 1.27.1
Kubernetes Version: 1.31.9
Cloud Provider: AWS EKS
CNI: AWS VPC CNI
Mode: Ambient Mode
Network Constraints: Limited IP pool causing frequent IP address reuse

Issue Details

Current Behavior

1. I think that the waypoint proxy maintains half-open HBONE connections to ztunnel after the original pod is terminated.

• Through tcpdump analysis, i saw that the reused IP address receives HTTP/2 stream data frames intended for the previous connection

2. We observed that RST packets are sent immediately after the reused IP receives these HTTP/2 DATA frames
3. When a new pod is created with a recycled IP address (after 30s cooldown), it receives traffic intended for the previous pod
4. The new pod/ztunnel responds with RST packets as the connection state doesn't match
5. This results in 503 Upstream connection termination errors
6. Waypoint's retry mechanism (reset-before-request), i think, doesn't appear to handle this scenario effectively

Expected Behavior

Waypoint should detect stale connections and close them proactively (<- This seems related to ztunnel issue #1191 )
When receiving RST packets, waypoint should retry the request with a fresh connection

Help Wanted

1. root cause validation

• Are we correctly identifying this as a half-open connection issue?
• Could there be other factors contributing to this behavior?
• Has anyone else observed similar HTTP/2 DATA frames being sent to reused IPs?

2. mitigation strategies : i'm seeking guidance on the following potential solutions
   A. Retry Policy Adjustment : Would changing the retry policy from reset-before-request to reset resolve this issue?
   B. Keepalive Configuration : Can aggressive keepalive settings help detect and clean up half-open connections?

[howardjohn]

So there are two possible clients here, Ztunnel --> Ztunnel and Envoy --> Ztunnel. I'll start on the first path since I am more confident in the fine-detailed understanding there.

When a new connection is established, we look to find which pod to send it to, and then lookup if there are any connections already to the pod. Well, technically its keyed on dst-ip+Service account, but its not just IP.

When a new byte is sent on an existing connection, we send to the same destination.

I would expect getting a RST on the connection level to drop the HBONE connection, such that new bytes cannot come in, and new connections re-establish a new connection.

I am less sure that Envoy is as good at following this. It would be good to understand if these are new connections, or new data on existing connections.

[Dylan-KW]

Thank you for the quick response! Let me clarify the situation with more details:

Clarification on the connection path

To be clear, this issue is occurring in the Waypoint (Envoy) → Ztunnel path. (More specifically, it appears to be happening in HBONE connections established through the connect_originate cluster configuration in the waypoint's Envoy proxy.)

Testing Setup
We are repeatedly sending requests from the same curl client pod to test the service through waypoint:

[Curl Pod] → [Waypoint Proxy] → [Ztunnel] → [Destination Pod]
(same client)   (Envoy proxy)     (HBONE)     (changing target)

Test Sequence

Initial state: Curl pod sends requests → Waypoint → Pod-A (IP: 10.90.142.96) ✅ Success
Pod rotation: Pod-A is deleted, Pod-B is created and gets the same IP (10.90.142.96)
After rotation: Same curl pod continues sending requests → Waypoint → Pod-B ❌ 503 errors

Evidence from Waypoint logs

I'm attaching waypoint proxy logs that demonstrate the connection reuse issue:

Image 1: Initial connection creation for Pod-aaa(IP: 10.90.142.96) when request occurs.

This shows a new HBONE connection (ID: 7009) being established to Pod-aaa at IP 10.90.142.96.

Image 2: Connection reuse after Pod-aaa deletion and Pod-bbb creation with same IP

After Pod-aaa was deleted and Pod-bbb was created with the same IP (10.90.142.96), it seems like the waypoint proxy attempts to reuse the existing connection (ID: 79097) that was originally established to Pod-aaa.

[Stevenjin8]

We have http2 pings turned on for h2 connections, at least in between ztunnels,and pretty sure about waypoints. We can configure these via ztunnel env vars. Maybe setting more aggressive settings would be good since your addresses are getting reused.

Edit: this makes no sense. We'd have to edit these settings in envoy, no ztunnel.

[Dylan-KW]

Thank you for the clarification.

Symptom Mitigation via HTTP/2 Keepalive

Yes, i agree, more aggressive HTTP/2 keepalive settings could help mitigate the half-open connection reuse issue by detecting stale connections faster.

Thoughts on Potential Root Cause Resolution

Just brainstorming here, but I'm wondering about a couple of possible directions:
A. Connection Pool Key Enhancement (Just a thought)
I'm curious if Envoy's connection pooling could theoretically use more than just IP:PORT for connection keys?

What if the connection key could include additional metadata key-value?
If Envoy supports this (or could support it), maybe Istiod could inject such metadata?
This way, even when IPs are reused, existing connections wouldn't be reused because the connection pool would see them as completely different entries (different keys)
(not sure it is feasible with current Envoy architecture)

B. Connection State Synchronization
istio/ztunnel#1191 seems to be discussing similar connection lifecycle issues. If that gets resolved with better connection state management between waypoint and ztunnel, would that potentially address this issue too?

[radleywellsfs]

We also seem to be seeing this issue using the AWS VPC CNI and Istio on ambient mode with a waypoint in front of the workload pod. This always seems to correlate with when an old IP is re-used.

Version Info:
v1.32 EKS
v1.26.3 Istio (Ambient + Sidecars Dataplane) - only seeing this issue on ambient though.

Logs:
Initial pod with IP 10.208.21.119

2025-10-22T09:25:18.622171Z	debug	envoy connection external/envoy/source/common/network/connection_impl.cc:1092	[Tags: "ConnectionId":"4255"] connecting to 10.208.21.119:15008	thread=22

When a new pod is created with the same IP 10.208.21.119 we get:

2025-10-22T09:35:03.033051Z	debug	envoy connection external/envoy/source/common/tls/ssl_socket.cc:248	[Tags: "ConnectionId":"4255"] remote address:10.208.21.119:15008,TLS_error:|33554536:system library:OPENSSL_internal:Connection reset by peer:TLS_error_end|33554464:system library:OPENSSL_internal:Broken pipe:TLS_error_end	thread=22

Note that the ConnectionID is the same - however, we don't see the logging of the form using existing fully connected connection in this case.

An example waypoint error:

{
  "authority": "xxx.yyy.svc.cluster.local",
  "bytes_received": 0,
  "bytes_sent": 95,
  "connection_termination_details": null,
  "downstream_local_address": "172.20.199.168:80",
  "downstream_remote_address": "10.208.25.231:34640",
  "duration": 3,
  "method": "GET",
  "path": "/v0/my/path/blah?type=my-type",
  "principal": null,
  "protocol": "HTTP/1.1",
  "request_duration": 0,
  "request_id": "84bb8653-bbfe-4bee-92f8-c55bd745979e",
  "requested_server_name": null,
  "response_code": 503,
  "response_code_details": "upstream_reset_before_response_started{connection_termination}",
  "response_duration": null,
  "response_flags": "UC",
  "route_name": null,
  "source_environment": null,
  "start_time": "2025-10-21T16:08:10.450Z",
  "upstream_cluster": "inbound-vip|80|http|xxx.yyy.svc.cluster.local;",
  "upstream_host": "envoy://connect_originate/10.208.43.235:8080",
  "upstream_local_address": "envoy://internal_client_address/",
  "upstream_service_time": null,
  "upstream_transport_failure_reason": null,
  "user_agent": "Java-http-client/17.0.16",
  "x_forwarded_for": null
}

Debugging

• We have tried setting various KEEPALIVE ztunnel environment variables, but this did not help the situation e.g.,

KEEPALIVE_ENABLED=true
KEEPALIVE_TIME=15s
KEEPALIVE_INTERVAL=5s
KEEPALIVE_RETRIES=3
POOL_UNUSED_RELEASE_TIMEOUT=30s

• When updating the cooldown period of the VPC CNI to say 15 minutes and re-running our tests this did not reproduce. Not a real fix for us as we'll run out of available IPs, but does seem to confirm this being the problem.

[bmbferreira]

Also facing this issue on EKS using istio ambient (everything running on latest stable versions). After a lot of blood, sweat and tears, the only thing that worked for me was to add retries with an EnvoyFilter like this:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: listener-filters-edge
spec:
  workloadSelector:
    labels:
      gateway.networking.k8s.io/gateway-name: default-gateway
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
      patch:
        operation: MERGE
        value:
          name: "envoy.filters.network.http_connection_manager"
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
            stream_idle_timeout: 0s # Default 5 mins. Must be disabled for long-lived and streaming requests
            use_remote_address: true
            normalize_path: true
            merge_slashes: true
            path_with_escaped_slashes_action: UNESCAPE_AND_REDIRECT
            common_http_protocol_options:
              idle_timeout: 50s 
            request_timeout: 0s # disable overall request timeout to support long polls
    - applyTo: HTTP_ROUTE
      match:
        context: GATEWAY
        routeConfiguration:
          vhost:
            route:
              name: company.company-gateway.1
      patch:
        operation: MERGE
        value:
          route:
            retry_policy:
              num_retries: 15
              per_try_timeout: 15s
              host_selection_retry_max_attempts: 15
              retry_back_off:
                base_interval: 100ms
                max_interval: 1s
              retry_on: reset,connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes