[Bug] Temporal local env cannot be connected in unit tests

[eyal-ringwald-orchid]

What are you really trying to do?

this is a bug report raised after opening a support issue #11853

During our unit tests we run local env (as part of the beforeAll of a suite) of temporal to run the workflow on the service and those tests became flaky since on the worker initialization the worker could not connect to the new temporal env.

Describe the bug

the following code got an exception :
this.connection = await NativeConnection.connect(connectionOptions);

TransportError: tonic::transport::Error(Transport, ConnectError(ConnectError("tcp connect error", Os { code: 60, kind: TimedOut, message: "Operation timed out" })))
120 | this.options.temporalConfig,
121 | );

122 | this.connection = await NativeConnection.connect(connectionOptions);
| ^
123 | }
124 | return this.connection;
125 | }

the test looks like this :
this is a sample of the test beforeAll beforeEach afterAll and afterEach , the test basically runs a service with temporal worker and its beforeAll fails on starting up the worker since it does not connect.

beforeAll(async () => {
const port = await getFreePort();
originalEndpoint = process.env.TEMPORAL_ENDPOINT;
process.env.TEMPORAL_ENDPOINT = localhost:${port};
env = await TestWorkflowEnvironment.createLocal({
server: { namespace: 'default', port, ui: false },
});
})
beforeEach(async () => {

jest.clearAllMocks();
installSpy.mockImplementation();
getCreationCycleIdSpy = jest
  .spyOn(GetEntitiesUpdatesService.prototype, 'getCreationCycleId')
  .mockResolvedValue({
    [Entities.APPLICATIONS]: creationCycleId,
    [Entities.GROUPS]: creationCycleId,
    [Entities.POLICIES]: creationCycleId,
    [Entities.SERVICE_PRINCIPALS]: creationCycleId,
    [Entities.USERS]: creationCycleId,
  });

getAppsByIdsSpy = jest
  .spyOn(
    InitIntegrationForSpecificIdsService.prototype,
    'getAppsByIds',
  )
  .mockResolvedValue({
    dataPersistentResult: {
      failed: [
        { entityName: '1', id: '1', success: false },
        { entityName: '2', id: '2', success: false },
      ],
      originalNumberOfPayloads: 3,
      success: [{ entityName: '3', id: '3', success: true }],
    },
  });

....

const module: TestingModule = await Test.createTestingModule({
  imports: [AppModule],
}).compile();
app = module.createNestApplication();
await app.init();

});

afterEach(async () => {
await app?.close();
})

afterAll(async () => {
process.env.TEMPORAL_ENDPOINT = originalEndpoint;
await env?.teardown();
});

Minimal Reproduction

this is not reproducible in simple test , but only when a large set of suites running together. we run the jest tests in band for the service. it smells that the number of tests is the cause for the issue so it might be related to resources issue.
we have around 10 suites that running such tests with same pattern of running the local env in each suite.

Environment/Versions

• OS and processor: Apple M3 Pro sequia 15.3.2
• Temporal Version: 1.11.3
• Are you using Docker or Kubernetes or building Temporal from source? we use packages from npm. our services are deployed on docker in kubernetes , but the issue is about running a test in my development computer.

Additional context

we use nestjs.
the test does not fail when runing it singularily.
I tried to create the local env for each test as well but it did not help.
we use runInBand option to run the tests sequentially and not in parallel.
as a workaround i was advised by the support team to do retries until i can connect to the local env. it works.