Skip to content

Fix Papermill CI #212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Apr 25, 2025
+819 −480
Merged

Fix Papermill CI #212

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
5eb4734
Fix Papermill CI
Fokko Mar 19, 2025
bfbeeaa
Give this a try
Fokko Mar 19, 2025
3f2443d
pycolor
Fokko Mar 19, 2025
7db08b4
Second attempt
Fokko Mar 19, 2025
ef96020
I like this more
Fokko Mar 19, 2025
ad9b858
MOAR
Fokko Mar 19, 2025
497dbd9
Try this
Fokko Mar 19, 2025
6ce3f5e
Merge branch 'main' of github.com:databricks/docker-spark-iceberg int…
Fokko Apr 24, 2025
1a10877
Cleanup
Fokko Apr 24, 2025
9a48610
Cleanup
Fokko Apr 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 0 additions & 9 deletions spark/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,6 @@ RUN pip3 install -r requirements.txt
# Add scala kernel via spylon-kernel
RUN python3 -m spylon_kernel install

# Download and install IJava jupyter kernel
RUN curl https://github.com/SpencerPark/IJava/releases/download/v1.3.0/ijava-1.3.0.zip -Lo ijava-1.3.0.zip \
&& unzip ijava-1.3.0.zip \
&& python3 install.py --sys-prefix \
&& rm ijava-1.3.0.zip

# Optional env variables
ENV SPARK_HOME=${SPARK_HOME:-"/opt/spark"}
ENV PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.9.7-src.zip:$PYTHONPATH
Expand Down Expand Up @@ -77,9 +71,6 @@ RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2
&& rm awscliv2.zip \
&& rm -rf aws/

# Add iceberg spark runtime jar to IJava classpath
ENV IJAVA_CLASSPATH=/opt/spark/jars/*

RUN mkdir -p /home/iceberg/data \
&& curl https://data.cityofnewyork.us/resource/tg4x-b46p.json > /home/iceberg/data/nyc_film_permits.json \
&& curl https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2022-04.parquet -o /home/iceberg/data/yellow_tripdata_2022-04.parquet \
Expand Down
1 change: 1 addition & 0 deletions spark/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@

start-master.sh -p 7077
start-worker.sh spark://spark-iceberg:7077

start-history-server.sh
start-thriftserver.sh --driver-java-options "-Dderby.system.home=/tmp/derby"

Expand Down
Loading