How to enable STRICT PeerAuthentication (and AuthorizationPolicy) without blocking liveness and readiness checks using Istio ambient

[0xErnie]

I am a little bit puzzled because sometimes documentation is exclusively for the sidecar-mode, but not declared as such.

I am using Istio ambient 1.26.0 to secure a namespace including two deployments.

---
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: encryption-only
  namespace: ml-service
spec:
  mtls:
    mode: STRICT

All traffic should be encrypted, but as soon as I enable PeerAuthentication liveness and readiness probes are not able to reach the pod. Ist this intended behavior? How can I still use the probes while forcing all the other traffic to be encrypted?

Same with AuthorizationPolicy:

---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
 name: allow-api-to-ml
 namespace: ml-service
spec:
 selector:
   matchLabels:
     app: ml-service
 action: ALLOW
 rules:
 - from:
   - source:
       namespaces:
       - ml-service

I would like to restrict access to this service to the namespace. NetworkPolicy could be also used for this. After enabling this the probes or not probing anymore.

I tried adding an exception for the /health endpoint or the kubelets ip but still no probes.

I have found and read https://istio.io/latest/docs/ops/configuration/mesh/app-health-check/ but this seems to be a sidecar-only article.

I have found https://istio.io/latest/about/faq/security/#k8s-health-checks and probe rewrite still seems to be a sidecar-only thing.

[ilrudie]

The istio-cni plugin should be adding some iptables config which allows health probes. This doc on istio.io has a high level explanation of how it works and interacts with kube netpol.

By the sounds of your post, these rules aren't configured (or are being stripped perhaps?) resulting in kube health check traffic getting treated as "normal" inbound where STRICT peer auth causes it to be rejected.

[alexander-kauerz-at-adesso]

Thank you for this document!
I modified the Network Policy but still the probes are not delivered:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: service-api
  namespace: ml-service
spec:
  podSelector:
    matchLabels:
      app: service-api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: istio-ingress
  - from:
    - ipBlock:
        cidr: 169.254.7.127/32

[alexander-kauerz-at-adesso]

I deleted all NeworkPolicies for that Namespace, waited a bit and set the PeerAuthentication to STRICT:
Again all probes are failing and the pods get restarted.

It seems like Istio Is blocking the unencrypted traffic from the kubelet.

[ilrudie]

In a system functioning normally, kubelet traffic shouldn't go through our proxy. It's meant to be handled specially and to bypass the proxy so that STRICT mTLS can be enabled without impacting probes.

In Istio ambient, this problem is solved by using a combination of iptables rules and source network address translation (SNAT) to rewrite only packets that provably originate from the local node with a fixed link-local IP, so that they can be explicitly ignored by Istio policy enforcement as unsecured health probe traffic. A link-local IP was chosen as the default since they are typically ignored for ingress-egress controls, and by IETF standard are not routable outside of the local subnetwork.

In the node-local ztunnel can you see the traffic deny being logged?

[alexander-kauerz-at-adesso]

In the node-local ztunnel can you see the traffic deny being logged?

Yes, lots of lines like these:

2025-07-11T08:11:10.641930Z        error        access        connection complete        src.addr=172.16.0.177:49620 dst.addr=172.16.0.208:8001 dst.service="prediction-service-api.prediction-service.svc.cluster.local" dst.workload="prediction-service-api-77c8d54f7-xbr9g" dst.namespace="prediction-service" direction="inbound" bytes_sent=0 bytes_recv=0 duration="0ms" error="connection closed due to policy rejection: explicitly denied by: istio-system/istio_converted_static_strict"
2025-07-11T08:11:10.711877Z        error        access        connection complete        src.addr=172.16.0.65:22004 dst.addr=172.16.0.79:8000 dst.service="prediction-service-vetiver.prediction-service.svc.cluster.local" dst.workload="prediction-service-vetiver-6b85b49df5-rj6rf" dst.namespace="prediction-service" direction="inbound" bytes_sent=0 bytes_recv=0 duration="0ms" error="connection closed due to policy rejection: explicitly denied by: istio-system/istio_converted_static_strict"
2025-07-11T08:11:11.061654Z        error        access        connection complete        src.addr=172.16.0.129:16574 dst.addr=172.16.0.144:8001 dst.service="prediction-service-api.prediction-service.svc.cluster.local" dst.workload="prediction-service-api-77c8d54f7-ss5vm" dst.namespace="prediction-service" direction="inbound" bytes_sent=0 bytes_recv=0 duration="0ms" error="connection closed due to policy rejection: explicitly denied by: istio-system/istio_converted_static_strict"

[ilrudie]

The differing source addresses from what looks like your pod range suggest these are not health probes from kubelet. Seems like these are probably unmeshed pods on that node which are meant to be denied.