[Bug] TLS error from poll_workflow_task_queue #1106
Description
What are you really trying to do?
I'm installing Temporal with the Helm chart. Our Python workers start fine and connect to the Temporal server, but I'm getting a TLS error from poll_workflow_task_queue. I'm trying to determine what this code is trying to resolve in order to try to eliminate this error.
Describe the bug
Despite having TLS configured properly (I hope! — the clients connect fine), I'm getting a TLS error from the poll_workflow_task_queue. I have the long form of the K8S service name temporal-frontend.main.svc.cluster.local and the short name temporal-frontend.main.svc in the certificate SAN, and I am setting TEMPORAL_HOSTNAME=temporal-frontend.main.svc. Despite this, I'm getting the error message below about the TLS cert not being valid for IP 10.244.3.67. This is obviously the case because we're not gonna put ephemeral IPs into the SANs. This IP resolves to the Matching pod temporal-matching-6c8ff5f74c-gcwgc. So I'm trying to understand what the poll_workflow_task_queue is trying to resolve, and getting that IP address back, or how it's coming up with that IP address. It seems to me that it should be talking to temporal-frontend.main.svc, which would have the correct SAN, but it isn't?
Environment/Versions
- OS and processor: Ubuntu host
- Temporal Version: Python SDK
- Are you using Docker or Kubernetes or building Temporal from source? Kubernetes Helm chart v0.65.0
Additional context
Error message:
2025-09-15T17:17:43.248391Z WARN temporal_client::retry: gRPC call poll_workflow_task_queue retried 8 times error=Status { code: Unavailable, message: "connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, ::1, not 10.244.3.67\"", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None }