[Bug] SparkTBinaryFrontendService: java.net.BindException: Address already in use (Bind failed)

[Emor-nj]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

This error occurs occasionally when use beeline-mode in spark engine:

ERROR SparkTBinaryFrontendService: java.net.BindException: Address already in use (Bind failed)
Diagnostic: Failed to initialize SparkSQLEngine: Failed to initialize frontend service on xxx/xx.xx.xx.xx:0.
org.apache.kyuubi.KyuubiException: Failed to initialize SparkSQLEngine: Failed to initialize frontend service on xxx/xx.xx.xx.xx:0.
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:357)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:342)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:342)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:407)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:737)
Caused by: org.apache.kyuubi.KyuubiException: Failed to initialize frontend service on xxx/xx.xx.xx.xx:0.
	at org.apache.kyuubi.service.TBinaryFrontendService.initialize(TBinaryFrontendService.scala:122)
	at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
	at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
	at org.apache.kyuubi.service.Serverable.initialize(Serverable.scala:48)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.initialize(SparkSQLEngine.scala:70)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:353)
	... 10 more
Caused by: java.net.BindException: Address already in use (Bind failed)
	at java.net.PlainSocketImpl.socketBind(Native Method)
	at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:513)
	at java.net.ServerSocket.bind(ServerSocket.java:375)
	at java.net.ServerSocket.<init>(ServerSocket.java:237)
	at org.apache.kyuubi.service.TBinaryFrontendService.initialize(TBinaryFrontendService.scala:100)
	... 19 more

Affects Version(s)

1.9.2

Kyuubi Server Log Output

Kyuubi Engine Log Output

Kyuubi Server Configurations

Kyuubi Engine Configurations

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[pan3793]

Seems that socket bind does not guarantee to bind successfully even with port 0, maybe you can follow this topic to tune the OS configuration to mitigate the bind failure issue.

https://stackoverflow.com/questions/33813701/on-linux-when-performing-a-socket-bind-with-port-0-pick-a-random-port-using-c

Another idea is to enhance the Kyuubi's code to add a retry logic here, PR is welcome