Container built from 1.28.1 source can no longer startup on host system with proxy due to internal frontend connection failure

[christopher-watanabe-snkeos]

Expected Behavior

Updating from 1.27.1 to 1.28.1, a standalone, containerized Temporal service with postgresql persistence running on a host system with a proxy configured should be able to successfully start.

Actual Behavior

Updating from 1.27.1 to 1.28.1, internal clients in a standalone, containerized Temporal service are unable to establish connections to the internal frontend service.

The logs show the following warning:

{
   "level":"warn",
   "msg":"error creating sdk client",
   "service":"worker",
   "error":"failed reaching server: connection error: desc = \"transport: Error while dialing: failed to do connect handshake, response: \"HTTP/1.1 403 Forbidden\" ... The requested URL could not be retrieved. The following error was encountered while trying to retrieve the URL: 127.0.0.1:58322",
   "logging-call-at":"common/sdk/factory.go:98"
}

I've configured the NO_PROXY of the docker daemon to bypass the proxy for local connections inside the containers, so establishing a connection to the internal frontend service should be allowed. It could be a source of error if the proxy environment is not being respected in 1.28.1 due to code changes, maybe?

Steps to Reproduce the Problem

Reproduction is quite time-intensive given the complexity of the host system. However, setup would go something like:

1. Create a host system with a proxy, i.e. an AWS EC2 instance with outgoing security group rules that routes traffic through a proxy.
2. Containerize the temporal solution, building from version v1.28.1.
3. Setup a persistence server. I used postgres, but ultimately the error suggests that it's an internal connection that is disallowed by the proxy.
4. Run the dockerized service, ensuring that the environment is configured in such a way that temporal contacts the persistence server from step 3 and that the NO_PROXY environment variable lists any container-local and DB connections to proceed without being proxied (i.e. 127.0.0.1 and whatever hostname the DB can be contacted under).
5. Notice that the service fails to start because a connection to the internal frontend service cannot be established.

Specifications

• Version: v1.28.1
• Platform: MS Windows Server 2022 Standard

Additional Information

The only change to the container that I run the service in was that one is built from the 1.27.1 source and the other is built from the 1.28.1 source. I haven't dived deep into the original source yet, but my suspicion is that somewhere a grpc client is no longer configured to respect the proxy environment.

[christopher-watanabe-snkeos]

From having a look at the configuration template, I noticed there's a USE_INTERNAL_FRONTEND configuration param. When I set this to true, the error string in the description goes away, but then a different error is thrown when creating an sdk client for the worker:

{
   "level":"warn",
   "ts":"2025-09-09T15:54:14.314+0200",
   "msg":"error creating sdk client",
   "service":"worker",
   "error":"failed reaching server: context deadline exceeded","logging-call-at":"C:/temporal-server/common/sdk/factory.go:98"}

[ethan-gallant]

Not sure if I'm going crazy here but it seems the frontend no longer actually compiles due to some issues, I've been struggling to get our integrated server working accordingly.

../../../../../go/pkg/mod/go.temporal.io/server@v1.28.1/client/frontend/client.go:21:47: cannot use (*clientImpl)(nil) (value of type *clientImpl) as workflowservice.WorkflowServiceClient value in variable declaration: *clientImpl does not implement workflowservice.WorkflowServiceClient (missing method FetchWorkerConfig)
../../../../../go/pkg/mod/go.temporal.io/server@v1.28.1/client/frontend/client.go:35:9: cannot use &clientImpl{…} (value of type *clientImpl) as workflowservice.WorkflowServiceClient value in return statement: *clientImpl does not implement workflowservice.WorkflowServiceClient (missing method FetchWorkerConfig)
../../../../../go/pkg/mod/go.temporal.io/server@v1.28.1/client/frontend/metric_client.go:15:47: cannot use (*metricClient)(nil) (value of type *metricClient) as workflowservice.WorkflowServiceClient value in variable declaration: *metricClient does not implement workflowservice.WorkflowServiceClient (missing method FetchWorkerConfig)
../../../../../go/pkg/mod/go.temporal.io/server@v1.28.1/client/frontend/metric_client.go:29:9: cannot use &metricClient{…} (value of type *metricClient) as workflowservice.WorkflowServiceClient value in return statement: *metricClient does not implement workflowservice.WorkflowServiceClient (missing method FetchWorkerConfig)
../../../../../go/pkg/mod/go.temporal.io/server@v1.28.1/client/frontend/retryable_client.go:8:47: cannot use (*retryableClient)(nil) (value of type *retryableClient) as workflowservice.WorkflowServiceClient value in variable declaration: *retryableClient does not implement workflowservice.WorkflowServiceClient (missing method FetchWorkerConfig)
../../../../../go/pkg/mod/go.temporal.io/server@v1.28.1/client/frontend/retryable_client.go:18:9: cannot use &retryableClient{…} (value of type *retryableClient) as workflowservice.WorkflowServiceClient value in return statement: *retryableClient does not implement workflowservice.WorkflowServiceClient (missing method FetchWorkerConfig)

[bergundy]

@ethan-gallant the compilation is likely failing because you have a newer version of go.temporal.io/api than what the server version that you are working with supports.

[bergundy]

I'm not aware of any changes between 1.27 to 1.28 that would have changed the behavior but let me check internally with the team.

[ethan-gallant]

I'm not aware of any changes between 1.27 to 1.28 that would have changed the behavior but let me check internally with the team.

I figured it out, importing both the server and the client in the same project is a recipe for trouble. Needed to create two separate go projects for it to work (even though they share a codebase).

[christopher-watanabe-snkeos]

We had a look and tried to build from the most recent patch of 1.27, which is 1.27.3. This build encounters the same issue, so if it helps narrow the scope, 1.27.1 works but 1.27.3 does not. We didn't try 1.27.2, but the changelog between 1.27.2 to 1.27.3 only updates the version string and adds a comment, so I believe somewhere in the 1.27.1 to 1.27.2 changelog, there's an update that's affecting the internal grpc comms in the container when hosted on a system with a proxy.

Could it be related to the grpc library updates?

Let me know if there's any other tests I can run to yield more information about the problem!

[christopher-watanabe-snkeos]

An important point: We are setting the NO_PROXY env var of the container so as to allow all local comms within the container. This is a reason that I think that it must be a grpc client issue such that the client is not respecting the proxy environment.

[mathias-kogler-snkeos]

Looking at the go grpc library I found two issues, that could be the root cause:

• NO_PROXY environment variable no longer respected on custom resolver results grpc/grpc-go#8327 (fixed in v1.72.2)
• grpc.Dial fails to connect to unix domain socket when https_proxy is set grpc/grpc-go#8207 (fixed in 1.72.1)

Temporal 1.27.1 uses google.golang.org/grpc v1.70.0, which works
Temporal 1.27.2/v1.27.3 uses google.golang.org/grpc v1.71.0, which doesn't work
Temporal main uses google.golang.org/grpc v1.72.2, which already contains fixes for the two issues above.

[christopher-watanabe-snkeos]

Looking at the go grpc library I found two issues, that could be the root cause:

• NO_PROXY environment variable no longer respected on custom resolver results grpc/grpc-go#8327 (fixed in v1.72.2)
• grpc.Dial fails to connect to unix domain socket when https_proxy is set grpc/grpc-go#8207 (fixed in 1.72.1)

Temporal 1.27.1 uses google.golang.org/grpc v1.70.0, which works Temporal 1.27.2/v1.27.3 uses google.golang.org/grpc v1.71.0, which doesn't work Temporal main uses google.golang.org/grpc v1.72.2, which already contains fixes for the two issues above.

It is indeed the mentioned bugs that affect the proxy behavior!!! Great find. As a workaround, we apply a git patch to the version of the library prior to build so that we can still use the 1.28.1 release. If possible, we'd love to see the library updated in a future patch of 1.28 🙏🏼 Otherwise, we have a workaround and hope for a more permanent fix in 1.29 🤞🏼