[🐛 Bug]: Session is killed and removed from Session Map in the same second that it receives another request

[alequint]

What happened?

Session is killed and removed from Session Map in the same second that it receives another request.

My selenium tests are running in a Kubernetes cluster (OpenShift), and they are configured to point to a remote selenium grid deployed in the same cluster and installed via helm chart (using chrome node and with auto scale enabled). Follow versions and configuration details:

Versions

• selenium/hub = 4.17.0-20240123
• selenium/chrome-node = 4.17.0-20240123

AutoScaling as configured in Helm values

autoscaling:
  enabled: true
  scalingType: deployment
  scaledOptions:
    maxReplicaCount: 999

I have noticed the following behaviors:

• When my tests run one after the other they run fine till the end and pass
• When these same tests run all at the same time, sometimes (not always), hub lost a reference to the session before a request is processed resulting in NoSuchException error.

Here is the exception we have in the test container:

Session Id: 6e5053d746543857588f380cafbe73f1

org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 6e5053d746543857588f380cafbe73f1
Build info: version: '4.15.0', revision: '1d14b5521b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-372.59.1.el8_6.x86_64', java.version: '11.0.20.1'
Driver info: driver.version: unknown
Build info: version: '4.16.1', revision: '9b4c83354e'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-372.59.1.el8_6.x86_64', java.version: '11.0.17'
Driver info: org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$qHh51oC6$ByteBuddy$HdqocFzk
Command: [6e5053d746543857588f380cafbe73f1, quit {}]
Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 119.0.6045.159, chrome: {chromedriverVersion: 119.0.6045.105 (38c72552c5e..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:43971}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://admin:admin@selenium-h..., se:cdpVersion: 119.0.6045.159, se:vnc: ws://admin:admin@selenium-h..., se:vncEnabled: true, se:vncLocalAddress: ws://10.254.21.152:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
Session ID: 6e5053d746543857588f380cafbe73f1
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:200)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:133)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:52)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:191)
	at org.openqa.selenium.remote.TracedCommandExecutor.execute(TracedCommandExecutor.java:51)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:523)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:596)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:600)
	at org.openqa.selenium.remote.RemoteWebDriver.quit(RemoteWebDriver.java:438)
	at framework.AutomationBaseTest.shutdownDriver(AutomationBaseTest.java:464)
	at framework.AutomationBaseTest.afterClass(AutomationBaseTest.java:445)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
	at org.testng.internal.invokers.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:222)
	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:131)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.testng.TestRunner.privateRun(TestRunner.java:829)
	at org.testng.TestRunner.run(TestRunner.java:602)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
	at org.testng.TestNG.runSuites(TestNG.java:1099)
	at org.testng.TestNG.run(TestNG.java:1067)
	at org.testng.TestNG.privateMain(TestNG.java:1414)
	at org.testng.TestNG.main(TestNG.java:1378)
Number of open drivers: 0
FAILED: tests.testdriver.TestDriver.Driver
org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 6e5053d746543857588f380cafbe73f1
Build info: version: '4.15.0', revision: '1d14b5521b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-372.59.1.el8_6.x86_64', java.version: '11.0.20.1'
Driver info: driver.version: unknown
Build info: version: '4.16.1', revision: '9b4c83354e'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-372.59.1.el8_6.x86_64', java.version: '11.0.17'
Driver info: org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$qHh51oC6$ByteBuddy$HdqocFzk
Command: [6e5053d746543857588f380cafbe73f1, screenshot {}]
Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 119.0.6045.159, chrome: {chromedriverVersion: 119.0.6045.105 (38c72552c5e..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:43971}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://admin:admin@selenium-h..., se:cdpVersion: 119.0.6045.159, se:vnc: ws://admin:admin@selenium-h..., se:vncEnabled: true, se:vncLocalAddress: ws://10.254.21.152:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
Session ID: 6e5053d746543857588f380cafbe73f1
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:200)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:133)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:52)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:191)
	at org.openqa.selenium.remote.TracedCommandExecutor.execute(TracedCommandExecutor.java:51)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:523)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:596)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:600)
	at org.openqa.selenium.remote.RemoteWebDriver.getScreenshotAs(RemoteWebDriver.java:326)
	at framework.Logger.getScreenshot(Logger.java:687)
	at framework.Logger.getScreenshot(Logger.java:844)
	at tests.testdriver.TestDriver.Driver(TestDriver.java:851)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:677)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.testng.TestRunner.privateRun(TestRunner.java:829)
	at org.testng.TestRunner.run(TestRunner.java:602)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
	at org.testng.TestNG.runSuites(TestNG.java:1099)
	at org.testng.TestNG.run(TestNG.java:1067)
	at org.testng.TestNG.privateMain(TestNG.java:1414)
	at org.testng.TestNG.main(TestNG.java:1378)

Then here is what happens in Selenium Hub from the point 6e5053d746543857588f380cafbe73f1is created (03:04:48.747) till it has its slot released (03:18:12.023) and then deleted from Session Map (03:18:12.023). Few miliseconds later, in the same seconds (03:18:12.265), it complains saying it is Unable to process a request against the just removed session.

03:04:48.747 INFO [LocalDistributor.newSession] - Session created by the Distributor. Id: 6e5053d746543857588f380cafbe73f1 
 Caps: Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 119.0.6045.159, chrome: {chromedriverVersion: 119.0.6045.105 (38c72552c5e..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:43971}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: {}, se:bidiEnabled: false, se:cdp: ws://admin:admin@selenium-h..., se:cdpVersion: 119.0.6045.159, se:vnc: ws://admin:admin@selenium-h..., se:vncEnabled: true, se:vncLocalAddress: ws://10.254.21.152:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}

(...)

03:18:12.023 INFO [GridModel.release] - Releasing slot for session id 6e5053d746543857588f380cafbe73f1
03:18:12.023 INFO [LocalSessionMap.lambda$new$0] - Deleted session from local Session Map, Id: 6e5053d746543857588f380cafbe73f1
03:18:12.264 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request for an existing session: Unable to find session with ID: 6e5053d746543857588f380cafbe73f1
Build info: version: '4.15.0', revision: '1d14b5521b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-372.59.1.el8_6.x86_64', java.version: '11.0.20.1'
Driver info: driver.version: unknown
03:18:12.265 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 6e5053d746543857588f380cafbe73f1
Build info: version: '4.15.0', revision: '1d14b5521b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-372.59.1.el8_6.x86_64', java.version: '11.0.20.1'
Driver info: driver.version: unknown
	at org.openqa.selenium.grid.sessionmap.local.LocalSessionMap.get(LocalSessionMap.java:132)
	at org.openqa.selenium.grid.router.HandleSession.lambda$loadSessionId$4(HandleSession.java:160)
	at io.opentelemetry.context.Context.lambda$wrap$2(Context.java:224)
	at org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:122)
	at org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:382)
	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)
	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
	at org.openqa.selenium.grid.router.Router.execute(Router.java:87)
	at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
	at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)
	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)
	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)
	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)
	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
	at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:54)
	at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)
	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)
	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
	at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
	at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
	at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)
	at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
	at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)
	at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Later in the logs of hub it keeps trying to process that request and keeps complaining it is unable to find that session id.

Again, we do not see this problem when the test is executed in isolation. We can reproduce this when other tests are running at the same time (each test runs in a differenr container but all pointing to the hub remotely). Thoughts?

Complete hub log is attached for further analysis here:

selenium-hub-94b85cd4-b6bvs-selenium-hub.log

Command used to start Selenium Grid with Docker (or Kubernetes)

Selenium Grid is installed via helm as follows:


helm install -n {{ selenium_namespace }} selenium-grid docker-selenium/selenium-grid -f /tmp/selenium-values.yml --version 0.25.0

selenium-values-yml have versions and "autoscaling" mentioned above, but images are pulled from our internal registry. Follow template used (only chrome node is enabled):

autoscaling:
  enabled: true
  scalingType: deployment
  scaledOptions:
    maxReplicaCount: 999

chromeNode:
  imageName: {{ fvt_image_registry }}/selenium/node-chrome
  imageTag: 4.17.0-20240123
  imagePullSecret: artifactory

firefoxNode:
  enabled: false
  imageName: {{ fvt_image_registry }}/selenium/node-firefox
  imageTag: 4.17.0-20240123
  imagePullSecret: artifactory

edgeNode:
  enabled: false
  imageName: {{ fvt_image_registry }}/selenium/node-edge
  imageTag: 4.17.0-20240123
  imagePullSecret: artifactory

videoRecorder:
  imageName: {{ fvt_image_registry }}/selenium/video
  imageTag: ffmpeg-6.1-20240123
  imagePullSecret: artifactory

hub:
  imageName: {{ fvt_image_registry }}/selenium/hub
  imageTag: 4.17.0-20240123
  imagePullSecret: artifactory

ingress:
  hostname: selenium-grid.local
  path: /selenium

### Relevant log output

```shell
Client Side:

org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 6e5053d746543857588f380cafbe73f1
Build info: version: '4.15.0', revision: '1d14b5521b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-372.59.1.el8_6.x86_64', java.version: '11.0.20.1'
Driver info: driver.version: unknown
Build info: version: '4.16.1', revision: '9b4c83354e'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-372.59.1.el8_6.x86_64', java.version: '11.0.17'
Driver info: org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$qHh51oC6$ByteBuddy$HdqocFzk
Command: [6e5053d746543857588f380cafbe73f1, quit {}]
Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 119.0.6045.159, chrome: {chromedriverVersion: 119.0.6045.105 (38c72552c5e..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:43971}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://admin:admin@selenium-h..., se:cdpVersion: 119.0.6045.159, se:vnc: ws://admin:admin@selenium-h..., se:vncEnabled: true, se:vncLocalAddress: ws://10.254.21.152:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
Session ID: 6e5053d746543857588f380cafbe73f1
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:200)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:133)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:52)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:191)
	at org.openqa.selenium.remote.TracedCommandExecutor.execute(TracedCommandExecutor.java:51)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:523)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:596)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:600)
	at org.openqa.selenium.remote.RemoteWebDriver.quit(RemoteWebDriver.java:438)
	at framework.AutomationBaseTest.shutdownDriver(AutomationBaseTest.java:464)
	at framework.AutomationBaseTest.afterClass(AutomationBaseTest.java:445)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
	at org.testng.internal.invokers.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:222)
	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:131)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.testng.TestRunner.privateRun(TestRunner.java:829)
	at org.testng.TestRunner.run(TestRunner.java:602)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
	at org.testng.TestNG.runSuites(TestNG.java:1099)
	at org.testng.TestNG.run(TestNG.java:1067)
	at org.testng.TestNG.privateMain(TestNG.java:1414)
	at org.testng.TestNG.main(TestNG.java:1378)
Number of open drivers: 0
FAILED: tests.testdriver.TestDriver.Driver
org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 6e5053d746543857588f380cafbe73f1
Build info: version: '4.15.0', revision: '1d14b5521b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-372.59.1.el8_6.x86_64', java.version: '11.0.20.1'
Driver info: driver.version: unknown
Build info: version: '4.16.1', revision: '9b4c83354e'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-372.59.1.el8_6.x86_64', java.version: '11.0.17'
Driver info: org.openqa.selenium.remote.RemoteWebDriver$ByteBuddy$qHh51oC6$ByteBuddy$HdqocFzk
Command: [6e5053d746543857588f380cafbe73f1, screenshot {}]
Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 119.0.6045.159, chrome: {chromedriverVersion: 119.0.6045.105 (38c72552c5e..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:43971}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://admin:admin@selenium-h..., se:cdpVersion: 119.0.6045.159, se:vnc: ws://admin:admin@selenium-h..., se:vncEnabled: true, se:vncLocalAddress: ws://10.254.21.152:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
Session ID: 6e5053d746543857588f380cafbe73f1
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:200)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:133)
	at org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:52)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:191)
	at org.openqa.selenium.remote.TracedCommandExecutor.execute(TracedCommandExecutor.java:51)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:523)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:596)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:600)
	at org.openqa.selenium.remote.RemoteWebDriver.getScreenshotAs(RemoteWebDriver.java:326)
	at framework.Logger.getScreenshot(Logger.java:687)
	at framework.Logger.getScreenshot(Logger.java:844)
	at tests.testdriver.TestDriver.Driver(TestDriver.java:851)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:677)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.testng.TestRunner.privateRun(TestRunner.java:829)
	at org.testng.TestRunner.run(TestRunner.java:602)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
	at org.testng.TestNG.runSuites(TestNG.java:1099)
	at org.testng.TestNG.run(TestNG.java:1067)
	at org.testng.TestNG.privateMain(TestNG.java:1414)
	at org.testng.TestNG.main(TestNG.java:1378)



Hub:


03:04:48.747 INFO [LocalDistributor.newSession] - Session created by the Distributor. Id: 6e5053d746543857588f380cafbe73f1 
 Caps: Capabilities {acceptInsecureCerts: true, browserName: chrome, browserVersion: 119.0.6045.159, chrome: {chromedriverVersion: 119.0.6045.105 (38c72552c5e..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:43971}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: {}, se:bidiEnabled: false, se:cdp: ws://admin:admin@selenium-h..., se:cdpVersion: 119.0.6045.159, se:vnc: ws://admin:admin@selenium-h..., se:vncEnabled: true, se:vncLocalAddress: ws://10.254.21.152:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}

(...)

03:18:12.023 INFO [GridModel.release] - Releasing slot for session id 6e5053d746543857588f380cafbe73f1
03:18:12.023 INFO [LocalSessionMap.lambda$new$0] - Deleted session from local Session Map, Id: 6e5053d746543857588f380cafbe73f1
03:18:12.264 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request for an existing session: Unable to find session with ID: 6e5053d746543857588f380cafbe73f1
Build info: version: '4.15.0', revision: '1d14b5521b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-372.59.1.el8_6.x86_64', java.version: '11.0.20.1'
Driver info: driver.version: unknown
03:18:12.265 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 6e5053d746543857588f380cafbe73f1
Build info: version: '4.15.0', revision: '1d14b5521b'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '4.18.0-372.59.1.el8_6.x86_64', java.version: '11.0.20.1'
Driver info: driver.version: unknown
	at org.openqa.selenium.grid.sessionmap.local.LocalSessionMap.get(LocalSessionMap.java:132)
	at org.openqa.selenium.grid.router.HandleSession.lambda$loadSessionId$4(HandleSession.java:160)
	at io.opentelemetry.context.Context.lambda$wrap$2(Context.java:224)
	at org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:122)
	at org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:382)
	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)
	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
	at org.openqa.selenium.grid.router.Router.execute(Router.java:87)
	at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
	at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)
	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)
	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)
	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)
	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
	at org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:54)
	at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)
	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:345)
	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
	at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
	at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
	at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)
	at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
	at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)
	at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Operating System

Kubernetes (OpenShift)

Docker Selenium version (image tag)

4.17.0-20240123

Selenium Grid chart version (chart version)

0.25.0

[github-actions]

@alequint, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[VietND96]

@diemol, have you seen something similar was reported to the upstream repo before? I guess there is no actionable from the container side.

[diemol]

No, I've never seen that reported. If it truly was an issue, several people would be affected.

I don't really know what to do with an issue like this. It feels like they are dump their environment logs here for us to figure out what the issue is.

[alequint]

@VietND96 @diemol the logs above are mainly to give you a clearer understanding of what issue we are facing... in summary, hub detects session was killed, then hub release the slot, delete the session and, in the same second we they say, correctly, to be unable to execute request for that just removed session.

In what situation NoSuchSessionException may happen in practice?

I don't know how test could run against a session that it just killed, may any async process trying to use a session no longer in use by accident as it happens in the same second when session was killed?

[diemol]

In what situation NoSuchSessionException may happen in practice?

When you are using a session id that is no longer active.

When my tests run one after the other they run fine till the end and pass

This is a good indicator that your test code is not working well when executed concurrently. Most likely you are using static variables or the like. If you need help figuring that out, feel free to join our chat where more people can help. I will leave the links for that below.

[github-actions]

💬 Please ask questions at:

• 📫 The Selenium user group
• 📮 StackOverflow
• 🗣 Our IRC/Slack/Matrix channels where the community can help you as well

[alequint]

Hey guys, @diemol @VietND96, I understand the issue is closed, but I would like to give you more context about this.

First, a comment from @diemol :

This is a good indicator that your test code is not working well when executed concurrently. Most likely you are using static variables or the like

This is not really the case, because we have separated containers running 1 different client for each test we run, so they are not sharing same variables of even environment, though both point to the same grid via remote driver.

Another comment from @diemol :

No, I've never seen that reported. If it truly was an issue, several people would be affected.

@diemol I have been looking at the backlog of issues we have here, and I noticed some similar issues being reported, quite similar to what we reported here. And in all the cases, however, users have difficulty to reproduce the same, pretty much like me, as they really happen randomly and there is no clear step by step to reach that bad result.

Let me point some of the issues we noticed here:

• [🐛 Bug]: selenium.NoSuchSessionException: Unable to find session with ID #2135 (very recent, 3 days ago)
• [🐛 Bug]: invalid session id after upgrading to 4.15 (helm-charts on k8s) #2039
• [🐛 Bug]: Deleted session from local Session Map #1922 (and here a second person points they are having the same problem, look at this comment here)

By the fact that several people are reporting the same issue and struggle to reproduce them clearly, isn't there anything or any insights or suggestions you guys could provide for cases like this?

Let me know if we can reopen the issue and continue the work here, we are really focused on that problem and can contribute to that issue others are also reporting

[diemol]

Can you share a test script we can use to reproduce the script reliably? With that, we'd be happy to reopen and have a look at this.

[VietND96]

In my view, looks like this unstable appears when autoscaling in K8s, even your report here also autoscaling.enabled: true, scalingType: deployment
I don't know that when this failure was happening, how many Node pods are up, and how many Node pods are being (or could be) terminated (by the scaler)
In the autoscaling type deployment, the Node pods can be scaled down randomly by the cool-down period, and to ensure the Node shutdown gracefully, in preStop hook there is cURL calling the Node drain endpoint, which is described here
I am thinking about a possible race condition that can happen, something like, in the same second (or the latency around 1-2 seconds in distributed components communication), for example, the Distributor already assigned a session to a Node (e.g just wrote to SessionMap and the signal not come to Node), the scaler also scaled down that Node, the Node send drain signal to Grid and it was accepted to drain immediately instead of stay with status draining.
That is my thought when seeing "in the same second", still need to find logs to prove that is the cause of this unstable