java.lang.IllegalAccessError when running with Spark 2.4.0

[amorskoy]

Before build I've changed spark version in pom.xml:
Spark-submit done in local[*] mode

<spark.version>2.4.0</spark.version>

java.lang.IllegalAccessError: tried to access class org.apache.spark.shuffle.sort.ShuffleInMemorySorter from class org.apache.spark.shuffle.sort.SplashUnsafeSorter
at org.apache.spark.shuffle.sort.SplashUnsafeSorter.(SplashUnsafeSorter.scala:77)
at org.apache.spark.shuffle.sort.SplashUnsafeShuffleWriter.(SplashUnsafeShuffleWriter.scala:56)
at org.apache.spark.shuffle.SplashShuffleManager.getWriter(SplashShuffleManager.scala:84)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:98)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

[jealous]

This error looks strange. These classes live in the same package. I check the code of spark 2.4.0, ShuffleInMemorySorter is a class with package-level visibility. We should be able to access this class.
I have also tried the tests with following command:

mvn package -Dspark.version=2.4.0

And it finishes successfully.
Would you please try the same command and see if you have any issues accessing this class?

[gmcaps]

Exception in thread "main" java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x2a265ea9) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x2a265ea9
at org.apache.spark.storage.StorageUtils$.(StorageUtils.scala:213)
at org.apache.spark.storage.BlockManagerMasterEndpoint.(BlockManagerMasterEndpoint.scala:114)
at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:353)
at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:290)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:339)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:194)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:279)
at org.apache.spark.SparkContext.(SparkContext.scala:464)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2704)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:953)
at scala.Option.getOrElse(Option.scala:201)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:947)
at SimpleApp$.main(SimpleApp.scala:6)
at SimpleApp.main(SimpleApp.scala)

Above error is coming while running spark application

[amareshb]

@gmcaps - I am facing a similar issue as well, were you able to resolve this?

[amareshb]

@gmcaps I was able to solve the issue you mentioned by adding JAVA_HOME to path env variables.

[zcattacz]

When trying to start thriftserver by spark-class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --hiveconf spark.master=spark://127.0.0.1:7077, I run into the exact same issue as @gmcaps mentioned, I also tried @amareshb 's suggestion to export PATH=$JAVA_HOME:$PATH without success.
My setup is OpenJDK 17.0.14 Headless on Debian 12, pySpark 3.5.4

Tried switching to msopenjdk11, the error disappeared.