How to bump spark version in elyra/kernel-spark-py?

[paf91]

Hi! Is there a simple way to bump spark and pyspark version in image elyra/kernel-spark-py:3.2.3?
Lower version (3.2) is not supported by the best spark clickhouse connector of latest version 0.7.3, Spark 3.4.2 Java 17 Scala 2.12 would be great or at least Spark 3.3.
https://housepower.github.io/spark-clickhouse-connector/#requirements

Looking at this Dockerfile
makes me think I can't just pass new spark version as hadoop2.7 is not included in repo, I need also to change binary to hadoop3
Of course that is not enough and I would like to get more libraries, like hadoop-aws:3.3.4 to get S3 support.

[paf91]

So basically I made Spark 3.4.2 and Spark 3.3.4 images on top of elyra/kernel-py:3.2.3. Created kernel.json based on spark_python_kubernetes kernel in EG 3.2.3.
As a result, Driver+executor pods are initialized, but sparksession for Spark 3.4.2 hangs on init stage and never initializes, meanwhile Spark 3.3.4 works just fine.
Something is wrong but can't figure out what it is, maybe the pod template? I simply copied EG 3.2.3 spark_python_kubernetes kernel folder and changed the image to what I built on top of elyra/kernel-py:3.2.3

[patilsanket48]

hi @paf91 , i am relative new to the spark in JEG, could please share the dockerfile that you used for the spark update ?

[lresende]

@paf91 and/or @patilsanket48 , you could update the Spark Version in the makefile and build new images for testing,

enterprise_gateway/Makefile

Line 17 in ffebbaf

VERSION?=3.3.0.dev0

And hopefully also provide a pr for the project

[patilsanket48]

i am trying to build it via makefile .. i think for python it is building but for spark with R it is not .. i am getting the the error as following

Dockerfile:119
--------------------
 118 |     
 119 | >>> RUN Rscript -e 'install.packages("IRkernel", repos="https://mirror.las.iastate.edu/CRAN/", lib="/opt/conda/lib/R/library")' \
 120 | >>>             -e 'IRkernel::installspec(prefix = "/usr/local")' \
 121 | >>>             -e 'download.file(url = "https://cran.r-project.org/src/contrib/Archive/SparkR/SparkR_3.1.2.tar.gz", destfile = "SparkR_3.1.2.tar.gz")' \
 122 | >>>             -e 'install.packages(pkgs="SparkR_3.1.2.tar.gz", type="source", repos=NULL, lib="/opt/conda/lib/R/library")' \
 123 | >>>             -e 'unlink("SparkR_3.1.2.tar.gz")'
 124 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c Rscript -e 'install.packages(\"IRkernel\", repos=\"https://mirror.las.iastate.edu/CRAN/\", lib=\"/opt/conda/lib/R/library\")'             -e 'IRkernel::installspec(prefix = \"/usr/local\")'             -e 'download.file(url = \"https://cran.r-project.org/src/contrib/Archive/SparkR/SparkR_3.1.2.tar.gz\", destfile = \"SparkR_3.1.2.tar.gz\")'             -e 'install.packages(pkgs=\"SparkR_3.1.2.tar.gz\", type=\"source\", repos=NULL, lib=\"/opt/conda/lib/R/library\")'             -e 'unlink(\"SparkR_3.1.2.tar.gz\")'" did not complete successfully: exit code: 1
make[1]: *** [Makefile:176: ../.image-demo-base] Error 1
make[1]: Leaving directory '/kroneNet/Userdata/homefolder/desle1/patils/enterprise_gateway/etc'
make: *** [Makefile:164: demo-base] Error 2

i think rspark is depreciated it may not have further support. let me know what should i raise for PR for specifically rspark