Skip to content

Commit 548b0e6

Browse files
lfranckedervoeti
lfrancke
and
dervoeti
authored
Updates Druid versions for 24.7 (#731)
* Updates Druid versions for 24.7 - Adds 30.0.0, removes 27.0.0 - Fixes run-druid script to include parameters needed for Java 11 and above - Switch to Java 17 for runtime (building still has an error, I'm on it) - Switches the Dockerfile to heredoc format for easier reading - Installs "native" pyyaml instead of via pip - Adds package_manifest - Removes outdated comment from base image - Fixes typo in upload script * Updates - Builds Prometheus Emitter from source and includes it directly - Adds patch to update patch level dependencies * Fix Changelog * Fix Changelog * Forgotten changes * Extend comment for dataformat-xml * Split the patches into four distinct ones and add them for every version we support. * Review comments - hadolint ignore instead of || exit - Add comment on why we try to shrink intermediate images * Remove Log4Shell workarounds. Druid 26.0.0 uses Log4j 2.18 which is not vulnerable anymore * Add dependencies required for XmlLayout to the build directly. * Add dependencies required for XmlLayout to the build directly * Merge changes * Fix bad permissions leading to test failures var/druid/metadata.db could not be created * Make build less noisy * Update labels * Updates - Add common labels - Update image-tools to 0.0.11 - Don't build targz for Druid * Updates - Add common labels - Update image-tools to 0.0.11 - Don't build targz for Druid * Remove accidentally committed line * Silence hadolint warning * Silence hadolint warning * Fix Docker lint warning * Add a build arg to keep the cache directories around for NPM, Maven and .cache * Update image-tools to 0.0.12 * Remove accidental commit * Update druid/Dockerfile Co-authored-by: Lukas Voetmand <lukas.voetmand@stackable.tech> * Delete duplicate changelog entry * Update druid/Dockerfile Co-authored-by: Lukas Voetmand <lukas.voetmand@stackable.tech> --------- Co-authored-by: Lukas Voetmand <lukas.voetmand@stackable.tech>
1 parent 6dfda14 commit 548b0e6

32 files changed

+1222
-119
lines changed

‎.github/actions/build/action.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ inputs:
1010
required: true
1111
image-tools-version:
1212
description: The Stackable image-tools version
13-
default: 0.0.10
13+
default: 0.0.12
1414
build-cache-nexus-password:
1515
description: Build cache password for the github user
1616
outputs:

‎.github/workflows/preflight.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@ jobs:
7070
- uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0
7171
with:
7272
python-version: '3.x'
73-
- run: pip install image-tools-stackabletech==0.0.8
73+
- run: pip install image-tools-stackabletech==0.0.12
7474
- name: Install preflight
7575
run: |
7676
wget https://github.com/redhat-openshift-ecosystem/openshift-preflight/releases/download/1.9.1/preflight-linux-amd64

‎.github/workflows/release.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ jobs:
6363
- name: Set up syft
6464
uses: anchore/sbom-action/download-syft@e8d2a6937ecead383dfe75190d104edd1f9c5751 # v0.16.0
6565
- name: Install image-tools-stackabletech
66-
run: pip install image-tools-stackabletech==0.0.8
66+
run: pip install image-tools-stackabletech==0.0.12
6767
- uses: docker/login-action@0d4c9c5ea7693da7b068278f7b52bda2a190a446 # v3.2.0
6868
with:
6969
registry: docker.stackable.tech

‎CHANGELOG.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ All notable changes to this project will be documented in this file.
3131
- trino & trino-cli: Add version 451 ([#758]).
3232
- airflow: Add version `2.8.4` and `2.9.2` ([#762]).
3333
- superset: Add version `3.1.3` and `4.0.2` ([#768]).
34+
- druid: Support for 30.0.0 using Java 17 ([#731])
3435

3536
### Changed
3637

@@ -77,6 +78,7 @@ All notable changes to this project will be documented in this file.
7778
- kafka: Remove unsupported version `3.5.2` ([#745]).
7879
- airflow: Remove unsupprted version `2.7.2`, `2.7.3` and `2.8.3` ([#762]).
7980
- superset: Remove version `2.1.1`, `3.0.1` and `3.0.3` ([#768]).
81+
- druid: Remove support for 27.0.0 ([#731])
8082

8183
[#583]: https://github.com/stackabletech/docker-images/pull/583
8284
[#611]: https://github.com/stackabletech/docker-images/pull/611
@@ -114,6 +116,7 @@ All notable changes to this project will be documented in this file.
114116
[#706]: https://github.com/stackabletech/docker-images/pull/706
115117
[#721]: https://github.com/stackabletech/docker-images/pull/721
116118
[#727]: https://github.com/stackabletech/docker-images/pull/727
119+
[#731]: https://github.com/stackabletech/docker-images/pull/731
117120
[#732]: https://github.com/stackabletech/docker-images/pull/732
118121
[#734]: https://github.com/stackabletech/docker-images/pull/734
119122
[#736]: https://github.com/stackabletech/docker-images/pull/736

‎druid/Dockerfile

Lines changed: 89 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -2,95 +2,111 @@
22

33
# Ignoring DL3038 globally because set `assumeyes=True` in dnf.conf in our base image
44
# Ignoring DL4006 globally because we inherit the SHELL from our base image
5-
# hadolint global ignore=DL3038,DL4006
5+
# Ignoring SC2164 globally because we inherit SHELL from the base image which contains "-e" for bash
6+
# hadolint global ignore=DL3038,DL4006,SC2164
67

78
# hadolint ignore=DL3006
8-
FROM stackable/image/java-devel as druid-builder
9+
FROM stackable/image/java-devel AS druid-builder
910

1011
ARG PRODUCT
1112
ARG JACKSON_DATAFORMAT_XML
1213
ARG STAX2_API
1314
ARG WOODSTOX_CORE
1415
ARG AUTHORIZER
1516

16-
RUN microdnf update && \
17-
microdnf install \
18-
# Required to install pyyaml
19-
python-pip \
20-
# Required to patch druid
21-
patch && \
22-
microdnf clean all && \
23-
rm -rf /var/cache/yum && \
24-
# pyyaml is required for the compile Druid
25-
pip install --no-cache-dir pyyaml==6.0.1
17+
# Setting this to anything other than "true" will keep the cache folders around (e.g. for Maven, NPM etc.)
18+
# This can be used to speed up builds when disk space is of no concern.
19+
ARG DELETE_CACHES="true"
20+
21+
RUN <<EOF
22+
microdnf update
23+
24+
# python-pyyaml:
25+
# This note was last checked for version 30.0.0
26+
# Required for the compilation of Druid.
27+
# This requirement is documented in docs/development/build.md and version 5.1 or later is required.
28+
# UBI 9 ships with 5.4.x so that should be fine
29+
#
30+
# patch: Required for the apply-patches.sh script
31+
microdnf install \
32+
python-pyyaml \
33+
patch
34+
35+
microdnf clean all
36+
rm -rf /var/cache/yum
37+
EOF
2638

2739
USER stackable
2840
WORKDIR /stackable
2941

3042
COPY --chown=stackable:stackable druid/stackable/patches/apply_patches.sh /stackable/apache-druid-${PRODUCT}-src/patches/apply_patches.sh
3143
COPY --chown=stackable:stackable druid/stackable/patches/${PRODUCT} /stackable/apache-druid-${PRODUCT}-src/patches/${PRODUCT}
3244

33-
RUN curl --fail -L "https://repo.stackable.tech/repository/packages/druid/apache-druid-${PRODUCT}-src.tar.gz" | tar -xzC . && \
34-
cd apache-druid-${PRODUCT}-src && \
35-
./patches/apply_patches.sh ${PRODUCT} && \
36-
mvn clean install -Pdist -pl '!extensions-core/druid-ranger-security' -DskipTests -Dmaven.javadoc.skip=true && \
37-
tar -xzf /stackable/apache-druid-${PRODUCT}-src/distribution/target/apache-druid-${PRODUCT}-bin.tar.gz && \
38-
mv /stackable/apache-druid-${PRODUCT}-src/apache-druid-${PRODUCT} /stackable/apache-druid-${PRODUCT} && \
39-
rm -rf /stackable/apache-druid-${PRODUCT}-src && \
40-
rm -rf /stackable/.m2 && \
41-
rm -rf /stackable/.npm && \
42-
rm -rf /stackable/.cache
43-
44-
# Do not remove the /stackable/apache-druid-${PRODUCT}/quickstart folder, it is needed for loading the Wikipedia
45-
# testdata in kuttl tests and the getting started guide.
46-
47-
# Install the Prometheus emitter extension. This bundle contains the emitter and all jar dependencies.
48-
RUN curl --fail "https://repo.stackable.tech/repository/packages/druid/druid-prometheus-emitter-${PRODUCT}.tar.gz" | tar -xzC /stackable/apache-druid-${PRODUCT}/extensions && \
49-
# Install OPA authorizer extension.
50-
curl --fail "https://repo.stackable.tech/repository/packages/druid/druid-opa-authorizer-${AUTHORIZER}.tar.gz" | tar -xzC /stackable/apache-druid-${PRODUCT}/extensions && \
51-
# Install jackson-dataformat-xml, stax2-api, and woodstox-core which are required for logging, and remove stax-ex.
52-
rm /stackable/apache-druid-${PRODUCT}/lib/stax-ex-*.jar && \
53-
curl --fail -L -o /stackable/apache-druid-${PRODUCT}/lib/jackson-dataformat-xml-${JACKSON_DATAFORMAT_XML}.jar \
54-
"https://repo.stackable.tech/repository/packages/jackson-dataformat-xml/jackson-dataformat-xml-${JACKSON_DATAFORMAT_XML}.jar" && \
55-
curl --fail -L -o /stackable/apache-druid-${PRODUCT}/lib/stax2-api-${STAX2_API}.jar \
56-
"https://repo.stackable.tech/repository/packages/stax2-api/stax2-api-${STAX2_API}.jar" && \
57-
curl --fail -L -o /stackable/apache-druid-${PRODUCT}/lib/woodstox-core-${WOODSTOX_CORE}.jar \
58-
"https://repo.stackable.tech/repository/packages/woodstox-core/woodstox-core-${WOODSTOX_CORE}.jar"
59-
60-
# For earlier versions this script removes the .class file that contains the
61-
# vulnerable code.
62-
# TODO: This can be restricted to target only versions which do not honor the environment
63-
# varible that has been set above but this has not currently been implemented
64-
COPY shared/log4shell.sh /bin
65-
RUN /bin/log4shell.sh "/stackable/apache-druid-${PRODUCT}"
66-
67-
# Ensure no vulnerable files are left over
68-
# This will currently report vulnerable files being present, as it also alerts on
69-
# SocketNode.class, which we do not remove with our scripts.
70-
# Further investigation will be needed whether this should also be removed.
71-
COPY shared/log4shell_1.6.1-log4shell_Linux_x86_64 /bin/log4shell_scanner_x86_64
72-
COPY shared/log4shell_1.6.1-log4shell_Linux_aarch64 /bin/log4shell_scanner_aarch64
73-
COPY shared/log4shell_scanner /bin/log4shell_scanner
74-
RUN /bin/log4shell_scanner s "/stackable/apache-druid-${PRODUCT}"
75-
# ===
45+
# Cache mounts are owned by root by default
46+
# We need to explicitly give the uid to use which is hardcoded to "1000" in stackable-base
47+
RUN --mount=type=cache,id=maven,uid=1000,target=/stackable/.m2/repository \
48+
--mount=type=cache,id=npm,uid=1000,target=/stackable/.npm \
49+
--mount=type=cache,id=cache,uid=1000,target=/stackable/.cache \
50+
<<EOF
51+
curl --fail -L "https://repo.stackable.tech/repository/packages/druid/apache-druid-${PRODUCT}-src.tar.gz" | tar -xzC .
52+
cd apache-druid-${PRODUCT}-src
53+
./patches/apply_patches.sh ${PRODUCT}
54+
55+
mvn --batch-mode --no-transfer-progress clean install -Pdist,stackable-bundle-contrib-exts -DskipTests -Dmaven.javadoc.skip=true
56+
mv distribution/target/apache-druid-${PRODUCT}-bin/apache-druid-${PRODUCT} /stackable/
57+
rm -rf /stackable/apache-druid-${PRODUCT}-src
58+
59+
# We're removing these to make the intermediate layer smaller
60+
# This can be necessary even though it's only a builder image because the GitHub Action Runners only have very limited space available
61+
# and we are sometimes running into errors because we're out of space.
62+
# Therefore, we try to clean up all layers as much as possible.
63+
if [ "${DELETE_CACHES}" = "true" ] ; then
64+
rm -rf /stackable/.m2/repository/*
65+
rm -rf /stackable/.npm/*
66+
rm -rf /stackable/.cache/*
67+
fi
68+
69+
# Do not remove the /stackable/apache-druid-${PRODUCT}/quickstart folder, it is needed for loading the Wikipedia
70+
# testdata in kuttl tests and the getting started guide.
71+
72+
# Install OPA authorizer extension.
73+
curl --fail -L "https://repo.stackable.tech/repository/packages/druid/druid-opa-authorizer-${AUTHORIZER}.tar.gz" | tar -xzC /stackable/apache-druid-${PRODUCT}/extensions
74+
EOF
7675

7776
# hadolint ignore=DL3006
78-
FROM stackable/image/java-base as final
77+
FROM stackable/image/java-base AS final
7978

8079
ARG PRODUCT
8180
ARG RELEASE
8281

83-
LABEL name="Apache Druid" \
84-
maintainer="info@stackable.tech" \
85-
vendor="Stackable GmbH" \
86-
version="${PRODUCT}" \
87-
release="${RELEASE}" \
88-
summary="The Stackable image for Apache Druid." \
89-
description="This image is deployed by the Stackable Operator for Apache Druid."
90-
91-
RUN microdnf update && \
92-
microdnf clean all && \
93-
rm -rf /var/cache/yum
82+
ARG NAME="Apache Druid"
83+
ARG DESCRIPTION="This image is deployed by the Stackable Operator for Apache Druid"
84+
85+
LABEL name="${NAME}"
86+
LABEL version="${PRODUCT}"
87+
LABEL release="${RELEASE}"
88+
LABEL summary="The Stackable image for Apache Druid"
89+
LABEL description="${DESCRIPTION}"
90+
91+
# https://github.com/opencontainers/image-spec/blob/036563a4a268d7c08b51a08f05a02a0fe74c7268/annotations.md#annotations
92+
LABEL org.opencontainers.image.documentation="https://docs.stackable.tech/home/stable/druid/"
93+
LABEL org.opencontainers.image.version="${PRODUCT}"
94+
LABEL org.opencontainers.image.revision="${RELEASE}"
95+
LABEL org.opencontainers.image.title="${NAME}"
96+
LABEL org.opencontainers.image.description="${DESCRIPTION}"
97+
98+
# https://docs.openshift.com/container-platform/4.16/openshift_images/create-images.html#defining-image-metadata
99+
# https://github.com/projectatomic/ContainerApplicationGenericLabels/blob/master/vendor/redhat/labels.md
100+
LABEL io.openshift.tags="ubi9,stackable,druid,sdp"
101+
LABEL io.k8s.description="${DESCRIPTION}"
102+
LABEL io.k8s.display-name="${NAME}"
103+
104+
RUN <<EOF
105+
microdnf update
106+
microdnf clean all
107+
rpm -qa --qf "%{NAME}-%{VERSION}-%{RELEASE}\n" | sort > /stackable/package_manifest.txt
108+
rm -rf /var/cache/yum
109+
EOF
94110

95111
USER stackable
96112
WORKDIR /stackable
@@ -99,9 +115,12 @@ COPY --chown=stackable:stackable --from=druid-builder /stackable/apache-druid-${
99115
COPY --chown=stackable:stackable druid/stackable/bin /stackable/bin
100116
COPY --chown=stackable:stackable druid/licenses /licenses
101117

102-
RUN ln -s /stackable/apache-druid-${PRODUCT} /stackable/druid && \
103-
# Force to overwrite the existing 'run-druid'
104-
ln -sf /stackable/bin/run-druid /stackable/druid/bin/run-druid
118+
RUN <<EOF
119+
ln -s /stackable/apache-druid-${PRODUCT} /stackable/druid
120+
121+
# Force to overwrite the existing 'run-druid'
122+
ln -sf /stackable/bin/run-druid /stackable/druid/bin/run-druid
123+
EOF
105124

106125
ENV PATH="${PATH}":/stackable/druid/bin
107126

‎druid/stackable/bin/run-druid

Lines changed: 32 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
#!/bin/bash -eu
22

3+
# This is adapted from the original `run-java` program from Druid 30.0.0 which was licensed under the Apache-2.0 license, as is this.
4+
# Source: https://github.com/apache/druid/blob/druid-30.0.0/examples/bin/run-java
5+
36
if [ "$#" -ne 2 ]
47
then
58
>&2 echo "usage: $0 <service> conf-dir"
@@ -9,12 +12,36 @@ fi
912
WHATAMI="$1"
1013
CONFDIR="$(cd "$2" && pwd)"
1114
# shellcheck disable=SC1091 # java-util is not available at this scripts source
12-
JAVA_BIN="$(source ./bin/java-util && get_java_bin_dir)"
15+
JAVA_BIN="$(source ./bin/java-util && get_java_bin_dir)/java"
1316
if [ -z "$JAVA_BIN" ]; then
1417
>&2 echo "Could not find java - please run ./bin/verify-java to confirm it is installed."
1518
exit 1
1619
fi
17-
# shellcheck disable=SC2046 # unsure how this `cat | xargs` works, and this script might rely on discouraged behaviour
18-
exec "$JAVA_BIN"/java $(cat "$CONFDIR"/jvm.config | xargs) \
19-
-cp "$CONFDIR":"./lib/*" \
20-
org.apache.druid.cli.Main server "$WHATAMI"
20+
21+
JAVA_MAJOR="$("$JAVA_BIN" -version 2>&1 | sed -n -E 's/.* version "([^."-]*).*/\1/p')"
22+
23+
if [ "$JAVA_MAJOR" != "" ] && [ "$JAVA_MAJOR" -ge "11" ]
24+
then
25+
# Disable strong encapsulation for certain packages on Java 11+.
26+
# When updating this list, update all four:
27+
# 1) ForkingTaskRunner#STRONG_ENCAPSULATION_PROPERTIES
28+
# 2) docs/operations/java.md, "Strong encapsulation" section
29+
# 3) pom.xml, jdk.strong.encapsulation.argLine
30+
# 4) examples/bin/run-java script (here)
31+
exec "$JAVA_BIN" \
32+
`cat "$CONFDIR"/jvm.config | xargs` \
33+
-cp "$CONFDIR":"./lib/*" \
34+
--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED \
35+
--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED \
36+
--add-opens=java.base/java.nio=ALL-UNNAMED \
37+
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED \
38+
--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED \
39+
--add-opens=java.base/java.io=ALL-UNNAMED \
40+
--add-opens=java.base/java.lang=ALL-UNNAMED \
41+
--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED \
42+
org.apache.druid.cli.Main server "$WHATAMI"
43+
else
44+
exec "$JAVA_BIN" `cat "$CONFDIR"/jvm.config | xargs` \
45+
-cp "$CONFDIR":"./lib/*" \
46+
org.apache.druid.cli.Main server "$WHATAMI"
47+
fi

‎druid/stackable/patches/26.0.0/01-remove-ranger-security.patch

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
Removes all traces of the druid ranger extension
2+
3+
From: Lars Francke <git@lars-francke.de>
4+
5+
6+
---
7+
0 files changed
8+
9+
diff --git a/distribution/pom.xml b/distribution/pom.xml
10+
index eec26171af..a6e72cf2c2 100644
11+
--- a/distribution/pom.xml
12+
+++ b/distribution/pom.xml
13+
@@ -255,8 +255,6 @@
14+
<argument>-c</argument>
15+
<argument>org.apache.druid.extensions:druid-pac4j</argument>
16+
<argument>-c</argument>
17+
- <argument>org.apache.druid.extensions:druid-ranger-security</argument>
18+
- <argument>-c</argument>
19+
<argument>org.apache.druid.extensions:druid-kubernetes-extensions</argument>
20+
<argument>-c</argument>
21+
<argument>org.apache.druid.extensions:druid-catalog</argument>
22+
@@ -439,8 +437,6 @@
23+
<argument>-c</argument>
24+
<argument>org.apache.druid.extensions:druid-pac4j</argument>
25+
<argument>-c</argument>
26+
- <argument>org.apache.druid.extensions:druid-ranger-security</argument>
27+
- <argument>-c</argument>
28+
<argument>org.apache.druid.extensions:druid-kubernetes-extensions</argument>
29+
<argument>${druid.distribution.pulldeps.opts}</argument>
30+
</arguments>
31+
diff --git a/pom.xml b/pom.xml
32+
index 0c6294f5ed..a33c6bd521 100644
33+
--- a/pom.xml
34+
+++ b/pom.xml
35+
@@ -186,7 +186,6 @@
36+
<module>extensions-core/simple-client-sslcontext</module>
37+
<module>extensions-core/druid-basic-security</module>
38+
<module>extensions-core/google-extensions</module>
39+
- <module>extensions-core/druid-ranger-security</module>
40+
<module>extensions-core/druid-catalog</module>
41+
<module>extensions-core/testing-tools</module>
42+
<!-- Community extensions -->

0 commit comments

Comments
 (0)