mutatingwebhookconfiguration not injecting to pods for clusters with no internet access

[kedai]

i'm testing irsa-manager on a cluster without internet access. I have http proxy configured.
issue - pods deployed with the annotation has no aws token mounted. seemed that the pod wasn't injected with the params.

kube-apiserver log

I0123 07:45:58.886461 1 trace.go:205] Trace[1636545974]: "Call mutating webhook" configuration:pod-identity-webhook,webhook:pod-identity-webhook.amazonaws.com,resource:/v1, Resource=pods,subresource:,operation:CREATE,UID:4ece82dd-c40b-4c34-afc3-9b6f51a89bb2 (23-Jan-2025 07:45:48.886) (total time: 10000ms):
Trace[1636545974]: [10.000309447s] [10.000309447s] END
W0123 07:45:58.886512 1 dispatcher.go:180] Failed calling webhook, failing open pod-identity-webhook.amazonaws.com: failed calling webhook "pod-identity-webhook.amazonaws.com": failed to call webhook: Post "https://pod-identity-webhook.kube-system.svc:443/mutate?timeout=10s": context deadline exceeded
E0123 07:45:58.886530 1 dispatcher.go:184] failed calling webhook "pod-identity-webhook.amazonaws.com": failed to call webhook: Post "https://pod-identity-webhook.kube-system.svc:443/mutate?timeout=10s": context deadline exceeded
I0123 07:45:58.891986 1 trace.go:205] Trace[860856684]: "Create" url:/api/v1/namespaces/default/pods,user-agent:kube-controller-manager/v1.25.10 (linux/amd64) kubernetes/e770bdb/system:serviceaccount:kube-system:replicaset-controller,audit-id:55fc7d9b-c23a-4e09-ba27-1e08af207872,client:10.236.46.150,accept:application/vnd.kubernetes.protobuf, /,protocol:HTTP/2.0 (23-Jan-2025 07:45:48.883) (total time: 10008ms):
Trace[860856684]: ---"Write to database call finished" len:485,err: 10007ms (07:45:58.891)
Trace[860856684]: [10.008075309s] [10.008075309s] END

kubectl get deployment -n kube-system pod-identity-webhook -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "4"
creationTimestamp: "2025-01-23T01:24:24Z"
generation: 4
name: pod-identity-webhook
namespace: kube-system
resourceVersion: "52784107"
uid: db812594-58a1-4740-815d-yy
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: pod-identity-webhook
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: pod-identity-webhook
spec:
containers:
- command:
- /webhook
- --in-cluster=false
- --namespace=kube-system
- --service-name=pod-identity-webhook
- --tls-secret=pod-identity-webhook
- --annotation-prefix=eks.amazonaws.com
- --token-audience=sts.amazonaws.com
- --logtostderr
env:
- name: HTTP_PROXY
value: http://x.x.x.x:yyyy
- name: HTTPS_PROXY
value: http://x.x.x.x:yyyy
- name: NO_PROXY
value: localhost,127.0.0.1,.svc.cluster.local,kubernetes.default.svc
- name: http_proxy
value: http://x.x.x.x:yyyy
- name: https_proxy
value: http://x.x.x.x:yyyy
- name: no_proxy
value: localhost,127.0.0.1,.svc.cluster.local,kubernetes.default.svc
image: amazon/amazon-eks-pod-identity-webhook:latest
imagePullPolicy: Always
name: pod-identity-webhook
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/webhook/certs
name: cert
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: pod-identity-webhook
serviceAccountName: pod-identity-webhook
terminationGracePeriodSeconds: 30
volumes:
- name: cert
secret:
defaultMode: 420
secretName: pod-identity-webhook
status:
availableReplicas: 1
conditions:

• lastTransitionTime: "2025-01-23T05:05:31Z"
  lastUpdateTime: "2025-01-23T05:05:36Z"
  message: ReplicaSet "pod-identity-webhook-7c4d95f498" has successfully progressed.
  reason: NewReplicaSetAvailable
  status: "True"
  type: Progressing
• lastTransitionTime: "2025-01-23T05:27:03Z"
  lastUpdateTime: "2025-01-23T05:27:03Z"
  message: Deployment has minimum availability.
  reason: MinimumReplicasAvailable
  status: "True"
  type: Available
  observedGeneration: 4
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

new pods deployed only have
k describe pod awscli-55c5df9df6-cmsmn
Name: awscli-55c5df9df6-cmsmn
Namespace: default
Priority: 0
Service Account: irsa-s3-limited-cu-sa
Node: some.svr/i.p.a.d
Start Time: Thu, 23 Jan 2025 13:29:47 +0800
Labels: app=awscli
pod-template-hash=55c5df9df6
Annotations: cni.projectcalico.org/containerID: edc83403d79554ed13831449048ce111b7319ec9da91514e01ebda76d787272f
cni.projectcalico.org/podIP: a.a.a.a/32
cni.projectcalico.org/podIPs: a.a.a.a/32
Status: Running
IP: a.a.a.a
IPs:
IP: a.a.a.a
Controlled By: ReplicaSet/awscli-55c5df9df6
Containers:
main:
Container ID: containerd://ea7c2ff7a4c91fb43890613c770bc2929d56762904546ab0a4c0944c588751ed
Image: amazon/aws-cli:latest
Image ID: docker.io/amazon/aws-cli@sha256:83ff20ed0625bdbbfad8f881069ebbd1ef4ee5b197b8fcf72489285862602257
Port:
Host Port:
Command:
/bin/sh
-c
sleep 36000
State: Running
Started: Thu, 23 Jan 2025 13:29:50 +0800
Ready: True
Restart Count: 0
Environment:
AWS_DEFAULT_REGION: some-region-1
ENABLE_IRP: true
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hgccf (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-hgccf:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 27m default-scheduler Successfully assigned default/awscli-55c5df9df6-cmsmn to some.svr
Normal Pulling 27m kubelet Pulling image "amazon/aws-cli:latest"
Normal Pulled 27m kubelet Successfully pulled image "amazon/aws-cli:latest" in 2.062437212s (2.062453768s including waiting)
Normal Created 27m kubelet Created container main
Normal Started 27m kubelet Started container main