Build Failure issues in Sunbird-Obsrv / sunbird-analytics-core (RELEASE-4.8.0) and Sunbird-Obsrv / sunbird-core-dataproducts (RELEASE-4.8.0) #114

sumanvpacewisdom · 2023-09-01T12:56:44Z

sumanvpacewisdom
Sep 1, 2023

I have the following system/software config

OS :- Ventura [Mac with M1 silicon]
Maven :- Apache Maven 3.9.4
Java :- openjdk 11.0.19 [Same version is used in IDE]
Scala :- Scala 3.3.0 [Scala 2.11.11 is used in IDE for sunbird-analytics-core AND Scala 2.11.8 is used in IDE for sunbird-core-dataproducts ]

Steps that I followed

Created a new folder called ml-exhaust
Cloned sunbird-analytics-core repo in ml-exhaust and checked out to the branch RELEASE-4.8.0

git clone https://github.com/Sunbird-Obsrv/sunbird-analytics-core.git
git checkout release-4.8.0

Built the entire code base from the location /ml-exhaust/sunbird-analytics-core via following command

mvn clean install -DskipTests

Result : BUILD FAILURE

Build Failure error

[ERROR] error: scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found.
[ERROR]         at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:17)
[ERROR]         at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:18)
[ERROR]         at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:53)
[ERROR]         at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:45)
[ERROR]         at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:45)
[ERROR]         at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:66)
[ERROR]         at scala.reflect.internal.Mirrors$RootsBase.getClassByName(Mirrors.scala:102)
[ERROR]         at scala.reflect.internal.Mirrors$RootsBase.getRequiredClass(Mirrors.scala:105)
[ERROR]         at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:257)
[ERROR]         at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:257)
[ERROR]         at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1390)
[ERROR]         at scala.tools.nsc.Global$Run.<init>(Global.scala:1242)
[ERROR]         at scala.tools.nsc.Driver.doCompile(Driver.scala:31)
[ERROR]         at scala.tools.nsc.MainClass.doCompile(Main.scala:23)
[ERROR]         at scala.tools.nsc.Driver.process(Driver.scala:51)
[ERROR]         at scala.tools.nsc.Driver.main(Driver.scala:64)
[ERROR]         at scala.tools.nsc.Main.main(Main.scala)
[ERROR]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR]         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR]         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR]         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[ERROR]         at scala_maven_executions.MainHelper.runMain(MainHelper.java:164)
[ERROR]         at scala_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Sunbird Analytics Framework 2.0:
[INFO] 
[INFO] Sunbird Analytics Framework ........................ SUCCESS [  0.512 s]
[INFO] analytics-core ..................................... FAILURE [ 10.900 s]
[INFO] Analytics Job Driver - Batch and Streaming ......... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  11.612 s
[INFO] Finished at: 2023-09-01T17:49:35+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (default) on project analytics-core: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (default) on project analytics-core: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:333)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: org.apache.maven.plugin.MojoExecutionException: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at scala_maven.ScalaMojoSupport.execute (ScalaMojoSupport.java:490)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:377)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:160)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:147)
    at scala_maven_executions.JavaMainCallerByFork.run (JavaMainCallerByFork.java:100)
    at scala_maven.ScalaCompilerSupport.compile (ScalaCompilerSupport.java:161)
    at scala_maven.ScalaCompilerSupport.doExecute (ScalaCompilerSupport.java:99)
    at scala_maven.ScalaMojoSupport.execute (ScalaMojoSupport.java:482)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :analytics-core

@manjudr Can you help me resolve this issue.

Thank you.

Cc- @vijiurs @aks30 @kiranharidas187 @Vivek-M-08 @rakeshSgr

Answered by manjudr

Sep 4, 2023

@sumanvpacewisdom - The release 4.8.0 works with spark 2.4.4. Please configure the run-job.sh script file to use the 2.4.4 spark version.

FYI - The error mentioned above suggests a conflict in Scala versions. Specifically, release 4.8.0 is built with Spark 2.4.4 with Scala 2.11. If you are using Spark 3.4 which is compatible with later of Scala (Scala 2.12/2.13).

View full answer

manjudr · 2023-09-04T05:54:07Z

manjudr
Sep 4, 2023
Maintainer

@sumanvpacewisdom - Which Java version are you currently using? release 4.8 is compatible with Java 8. If you would like to use Java 11, please make sure to use release 4.8.5 or later.

7 replies

sumanvpacewisdom Sep 4, 2023
Author

Thanks for the suggestion it helped me build the service successfully, While running the command

nohup /Users/sumanv/Documents/ShikshaLokam/Projects/ml-exhaust/sunbird-core-dataproducts/scripts/shell/local/local-run.sh druid-dataset

I am getting the following error

Starting the job - druid-dataset
Job modelName - druid-dataset
23/09/04 15:30:04 WARN Utils: Your hostname, PWA166.local resolves to a loopback address: 127.94.0.2; using 192.168.0.101 instead (on interface en0)
23/09/04 15:30:04 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
23/09/04 15:30:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoClassDefFoundError: scala/Product$class
	at optional.Options.<init>(options.scala:6)
	at optional.Options$.parse(options.scala:71)
	at optional.Application$class.main(application.scala:257)
	at org.ekstep.analytics.job.JobExecutor$.main(JobExecutor.scala:8)
	at org.ekstep.analytics.job.JobExecutor.main(JobExecutor.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: scala.Product$class
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 17 more
23/09/04 15:30:05 INFO ShutdownHookManager: Shutdown hook called
23/09/04 15:30:05 INFO ShutdownHookManager: Deleting directory /private/var/folders/0w/4_s7903s36d4jbhnxl0yvvzw0000gn/T/spark-b7288c05-8200-430e-aab6-f58fbc6cceeb
Job execution completed - druid-dataset

manjudr Sep 4, 2023
Maintainer

@sumanvpacewisdom - Please share these details -

What release tag is used for both analytics core and core-dataproducts?
What version of spark are you using in your run-job.sh?
What Java version is configured?

sumanvpacewisdom Sep 4, 2023
Author

Sunbird-analytics core - RELEASE 4.8.0 and Sunbird-core-dataproducts - RELEASE 4.8.0
Spark version - spark-3.4.1
JAVA configuration :
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)

manjudr Sep 4, 2023
Maintainer

@sumanvpacewisdom - The release 4.8.0 works with spark 2.4.4. Please configure the run-job.sh script file to use the 2.4.4 spark version.

FYI - The error mentioned above suggests a conflict in Scala versions. Specifically, release 4.8.0 is built with Spark 2.4.4 with Scala 2.11. If you are using Spark 3.4 which is compatible with later of Scala (Scala 2.12/2.13).

Answer selected by manjudr

manjudr Sep 4, 2023
Maintainer

You can download the spark 2.4.4 from here - https://archive.apache.org/dist/spark/spark-2.4.4/

manjudr Sep 4, 2023
Maintainer

Also, please refer to this spark 2.4.4 provision documents which required additional jars as well. Please download all these jars that are mentioned in the script and keep them under the spark jars directory - https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/blob/release-4.8.0/ansible/roles/analytics-spark-provision/defaults/main.yml#L28-L34

sumanvpacewisdom Sep 19, 2023
Author

This Issue is resolved .

@manjudr Thank you for solving my issue this helped me to run the job needed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sunbird Obsrv

Build Failure issues in Sunbird-Obsrv / sunbird-analytics-core (RELEASE-4.8.0) and Sunbird-Obsrv / sunbird-core-dataproducts (RELEASE-4.8.0) #114

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 7 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Sunbird Obsrv

Build Failure issues in Sunbird-Obsrv / sunbird-analytics-core (RELEASE-4.8.0) and Sunbird-Obsrv / sunbird-core-dataproducts (RELEASE-4.8.0) #114

Uh oh!

sumanvpacewisdom Sep 1, 2023

I have the following system/software config

Steps that I followed

Build Failure error

Replies: 1 comment · 7 replies

Uh oh!

manjudr Sep 4, 2023 Maintainer

Uh oh!

sumanvpacewisdom Sep 4, 2023 Author

Uh oh!

manjudr Sep 4, 2023 Maintainer

Uh oh!

sumanvpacewisdom Sep 4, 2023 Author

Uh oh!

manjudr Sep 4, 2023 Maintainer

Uh oh!

manjudr Sep 4, 2023 Maintainer

Uh oh!

manjudr Sep 4, 2023 Maintainer

Uh oh!

sumanvpacewisdom Sep 19, 2023 Author

sumanvpacewisdom
Sep 1, 2023

Replies: 1 comment 7 replies

manjudr
Sep 4, 2023
Maintainer

sumanvpacewisdom Sep 4, 2023
Author

manjudr Sep 4, 2023
Maintainer

sumanvpacewisdom Sep 4, 2023
Author

manjudr Sep 4, 2023
Maintainer

manjudr Sep 4, 2023
Maintainer

manjudr Sep 4, 2023
Maintainer

sumanvpacewisdom Sep 19, 2023
Author