Missing py4j dependency when trying to import pyspark from inside the container

[theoctober19th]

Upon running Python3 shell inside the container spawned from Charmed Spark Rock image, the import pyspark fails, with a missing py4j dependency.

Steps to Reproduce:

docker pull ghcr.io/canonical/charmed-spark:3.5.1-22.04_edge
docker run -d ghcr.io/canonical/charmed-spark:3.5.1-22.04_edge
docker container exec -it <container_name> bash

python3
>>> import pyspark

Observed Error:

>>> import pyspark
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/spark/python/pyspark/__init__.py", line 58, in <module>
    from pyspark.conf import SparkConf
  File "/opt/spark/python/pyspark/conf.py", line 23, in <module>
    from py4j.java_gateway import JVMView, JavaObject
ModuleNotFoundError: No module named 'py4j'

PS, this was found in both base Spark image as well as Jupyterlab image, in versions 3.4 as well as 3.5.

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-6133.

This message was autogenerated

[Batalex]

py4j is vendored in the spark distribution as a zip file. In charmed-spark, it is located under /opt/spark/python/lib. We can import it by adding its path to PYTHONPATH as follows:

PYTHONPATH=$PYTHONPATH:/opt/spark/python/lib/py4j-0.10.9.7-src.zip python3
Python 3.10.12 (main, Aug 15 2025, 14:32:43) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyspark
>>> import py4j
>>> py4j.__version__
'0.10.9.7'
>>> py4j.__file__
'/opt/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/__init__.py'

Therefore, we can add this entry to the PYTHONPATH definition in `rockcraft.yaml