Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.execution.auron.shuffle.AuronShuffleManager

[Dormant7]

i use auron in spark 。i submit spark job use : auron-spark-3.4_2.12-pre-osx-x86_64-6.0.0-incubating.jar to driver and executor config, but got an error:

25/10/31 07:47:08 INFO SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users with view permissions: root; groups with view permissions: EMPTY; users with modify permissions: root; groups with modify permissions: EMPTY
25/10/31 07:47:08 INFO TransportClientFactory: Successfully created connection to spark-temp-id-b99b65c508-sql-b68ad69a393551c1-driver-svc.c2aas-prod.svc/10.244.1.26:7078 after 55 ms (36 ms spent in bootstraps)
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:62)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:428)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:417)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.execution.auron.shuffle.AuronShuffleManager
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
at org.apache.spark.util.Utils$.instantiateSerializerOrShuffleManager(Utils.scala:2717)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:323)
at org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:212)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$7(CoarseGrainedExecutorBackend.scala:477)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:63)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
... 4 more

how to fix it , thank you。

[merrily01]

It looks like you’re using a pre-release (pre) build, which doesn’t include all the required classes ?

Maybe you can try building the release version instead — for example :

  ./auron-build.sh --clean true --skiptests true --release --sparkver 3.4 --scalaver 2.12

Additionally, have you already placed the Auron JAR package in the Spark jars path?

You can find the detailed usage instructions at https://auron.apache.org/documents/getting-started.html#run-spark-job-with-auron-accelerator.

[Dormant7]

thanks , I've solved this problem。
i have anthor question. How to verify that shuffle is using Auron?