How to successfully execute a spark job locally?

[AdrianoNicolucci]

I don't think the setup instructions to run a job locally are complete. I'm trying to execute a glue job locally...all it says is:

5. now you can run the executables in the bin directory to start a Glue Shell or submit a Glue Spark application.

Glue shell: ./bin/gluepyspark
Glue submit: ./bin/gluesparksubmit
pytest: ./bin/gluepytest

If I run ./bin/gluesparksubmit in bash followed by the name of the python file, it seems to successfully build followed by this error message and nothing happens:

mkdir: cannot create directory ‘/f/programming/aws-glue-libs/conf’: File exists
/f/programming/aws-glue-libs
Aug 20, 2023 4:58:21 P.M. org.apache.spark.launcher.Log4jHotPatchOption staticJavaAgentOption
WARNING: spark.log4jHotPatch.enabled is set to true, but /usr/share/log4j-cve-2021-44228-hotpatch/jdk17/Log4jHotPatchFat.jar does not exist at the configured location

"C:\Program Files\Java\jdk-20\bin\java" -cp "F:programmingspark/jars/*:/f/programming/aws-glue- libs/jarsv1/*;F:/programming/aws-glue-libs/conf\;F:\programming\\spark\jars\*" -Xmx1g -XX:+IgnoreUnrecognizedVMOptions "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" org.apache.spark.deploy.SparkSubmit --py-files F:/programming/aws-glue-libs/PyGlue.zip F:/programming//run_pyspark_locally.py
F:\programming\spark/bin/spark-class: line 97: CMD: bad array subscript

Can someone provide the actual parameters we need to pass to execute a job or an example how they have successfully used this library locally?

[nrajora]

Did you find an answer to this?