Skip to content

Commit 4655b4b

Browse files
jameslambjameslamb
authored
use 'rapids-init-pip' in wheel CI, other CI changes (#18902)
Proposes a batch of small, mostly-unrelated changes to building and packaging. These are all part of RAPIDS-wide efforts to make it easier to reproduce CI locally and to use artifacts from one project's CI in another project's CI. Contributes to rapidsai/build-planning#178 * prevents triggering expensive CI jobs based on PRs modifying `ci/release/update-version.sh` Contributes to rapidsai/shared-workflows#356 * explicitly provides an input for `script` to workflows using it, instead of relying on a default value coming from the workflow file Contributes to rapidsai/shared-workflows#337 * switches from input `run_script` to `script` in uses of the `custom-job` workflow Contributes to rapidsai/build-planning#179 * adds a call to `rapids-init-pip` near the beginning of all CI scripts that use `pip` * removed redefinitions of `PIP_CONSTRAINT` in scripts, in favor of using the one set up by `rapids-pip-init` Contributes to rapidsai/gha-tools#145 * confirmed that `rapids-configure-conda-channels` is not used in any builds scripts using `rattler-build` ## Notes for Reviewers # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #18902
1 parent 5625009 commit 4655b4b

15 files changed

+81
-24
lines changed

‎.github/workflows/build.yaml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ jobs:
4545
build_type: ${{ inputs.build_type || 'branch' }}
4646
branch: ${{ inputs.branch }}
4747
date: ${{ inputs.date }}
48+
script: ci/build_cpp.sh
4849
sha: ${{ inputs.sha }}
4950
node_type: "cpu16"
5051
python-build:
@@ -55,6 +56,7 @@ jobs:
5556
build_type: ${{ inputs.build_type || 'branch' }}
5657
branch: ${{ inputs.branch }}
5758
date: ${{ inputs.date }}
59+
script: ci/build_python.sh
5860
sha: ${{ inputs.sha }}
5961
upload-conda:
6062
needs: [cpp-build, python-build]
@@ -77,7 +79,7 @@ jobs:
7779
container_image: "rapidsai/ci-conda:latest"
7880
date: ${{ inputs.date }}
7981
node_type: "gpu-l4-latest-1"
80-
run_script: "ci/build_docs.sh"
82+
script: "ci/build_docs.sh"
8183
sha: ${{ inputs.sha }}
8284
wheel-build-libcudf:
8385
needs: [telemetry-setup]

‎.github/workflows/pr.yaml

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,7 @@ jobs:
8080
- '!CONTRIBUTING.md'
8181
- '!README.md'
8282
- '!ci/cudf_pandas_scripts/**'
83+
- '!ci/release/update-version.sh'
8384
- '!docs/**'
8485
- '!img/**'
8586
- '!java/**'
@@ -90,6 +91,7 @@ jobs:
9091
- '!.devcontainer/**'
9192
- '!CONTRIBUTING.md'
9293
- '!README.md'
94+
- '!ci/release/update-version.sh'
9395
- '!docs/**'
9496
- '!img/**'
9597
- '!java/**'
@@ -100,6 +102,7 @@ jobs:
100102
- '!CONTRIBUTING.md'
101103
- '!README.md'
102104
- '!ci/cudf_pandas_scripts/**'
105+
- '!ci/release/update-version.sh'
103106
- '!docs/**'
104107
- '!img/**'
105108
- '!notebooks/**'
@@ -110,13 +113,15 @@ jobs:
110113
- '!CONTRIBUTING.md'
111114
- '!README.md'
112115
- '!ci/cudf_pandas_scripts/**'
116+
- '!ci/release/update-version.sh'
113117
- '!java/**'
114118
test_python:
115119
- '**'
116120
- '!.devcontainer/**'
117121
- '!CONTRIBUTING.md'
118122
- '!README.md'
119123
- '!ci/cudf_pandas_scripts/**'
124+
- '!ci/release/update-version.sh'
120125
- '!docs/**'
121126
- '!img/**'
122127
- '!java/**'
@@ -135,13 +140,14 @@ jobs:
135140
with:
136141
build_type: pull-request
137142
node_type: "cpu16"
143+
script: ci/build_cpp.sh
138144
cpp-linters:
139145
secrets: inherit
140146
needs: checks
141147
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.06
142148
with:
143149
build_type: pull-request
144-
run_script: "ci/cpp_linters.sh"
150+
script: "ci/cpp_linters.sh"
145151
node_type: "cpu16"
146152
conda-cpp-checks:
147153
needs: conda-cpp-build
@@ -157,12 +163,14 @@ jobs:
157163
if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_cpp
158164
with:
159165
build_type: pull-request
166+
script: ci/test_cpp.sh
160167
conda-python-build:
161168
needs: conda-cpp-build
162169
secrets: inherit
163170
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-25.06
164171
with:
165172
build_type: pull-request
173+
script: ci/build_python.sh
166174
conda-python-cudf-tests:
167175
needs: [conda-python-build, changed-files]
168176
secrets: inherit
@@ -190,7 +198,7 @@ jobs:
190198
node_type: "gpu-l4-latest-1"
191199
arch: "amd64"
192200
container_image: "rapidsai/ci-conda:latest"
193-
run_script: "ci/test_java.sh"
201+
script: "ci/test_java.sh"
194202
conda-notebook-tests:
195203
needs: [conda-python-build, changed-files]
196204
secrets: inherit
@@ -201,7 +209,7 @@ jobs:
201209
node_type: "gpu-l4-latest-1"
202210
arch: "amd64"
203211
container_image: "rapidsai/ci-conda:latest"
204-
run_script: "ci/test_notebooks.sh"
212+
script: "ci/test_notebooks.sh"
205213
docs-build:
206214
needs: conda-python-build
207215
secrets: inherit
@@ -211,7 +219,7 @@ jobs:
211219
node_type: "gpu-l4-latest-1"
212220
arch: "amd64"
213221
container_image: "rapidsai/ci-conda:latest"
214-
run_script: "ci/build_docs.sh"
222+
script: "ci/build_docs.sh"
215223
wheel-build-libcudf:
216224
needs: checks
217225
secrets: inherit
@@ -337,7 +345,7 @@ jobs:
337345
node_type: "gpu-l4-latest-1"
338346
continue-on-error: true
339347
container_image: "rapidsai/ci-conda:cuda12.9.0-ubuntu24.04-py3.12"
340-
run_script: |
348+
script: |
341349
ci/cudf_pandas_scripts/third-party-integration/test.sh python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml
342350
pandas-tests:
343351
# run the Pandas unit tests using PR branch
@@ -359,7 +367,7 @@ jobs:
359367
with:
360368
node_type: "cpu4"
361369
build_type: pull-request
362-
run_script: "ci/cudf_pandas_scripts/pandas-tests/diff.sh"
370+
script: "ci/cudf_pandas_scripts/pandas-tests/diff.sh"
363371
narwhals-tests:
364372
needs: [conda-python-build, changed-files]
365373
secrets: inherit
@@ -372,7 +380,7 @@ jobs:
372380
sha: ${{ inputs.sha }}
373381
node_type: "gpu-l4-latest-1"
374382
container_image: "rapidsai/ci-conda:latest"
375-
run_script: ci/test_narwhals.sh
383+
script: ci/test_narwhals.sh
376384
spark-rapids-jni:
377385
needs: changed-files
378386
uses: ./.github/workflows/spark-rapids-jni.yaml

‎.github/workflows/test.yaml

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ jobs:
3333
build_type: ${{ inputs.build_type }}
3434
branch: ${{ inputs.branch }}
3535
date: ${{ inputs.date }}
36+
script: ci/test_cpp.sh
3637
sha: ${{ inputs.sha }}
3738
conda-cpp-memcheck-tests:
3839
secrets: inherit
@@ -45,7 +46,7 @@ jobs:
4546
node_type: "gpu-l4-latest-1"
4647
arch: "amd64"
4748
container_image: "rapidsai/ci-conda:latest"
48-
run_script: "ci/test_cpp_memcheck.sh"
49+
script: "ci/test_cpp_memcheck.sh"
4950
cpp-linters:
5051
secrets: inherit
5152
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.06
@@ -54,7 +55,7 @@ jobs:
5455
branch: ${{ inputs.branch }}
5556
date: ${{ inputs.date }}
5657
sha: ${{ inputs.sha }}
57-
run_script: "ci/cpp_linters.sh"
58+
script: "ci/cpp_linters.sh"
5859
file_to_upload: iwyu_results.txt
5960
conda-python-cudf-tests:
6061
secrets: inherit
@@ -86,7 +87,7 @@ jobs:
8687
node_type: "gpu-l4-latest-1"
8788
arch: "amd64"
8889
container_image: "rapidsai/ci-conda:latest"
89-
run_script: "ci/test_java.sh"
90+
script: "ci/test_java.sh"
9091
conda-notebook-tests:
9192
secrets: inherit
9293
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.06
@@ -98,7 +99,7 @@ jobs:
9899
node_type: "gpu-l4-latest-1"
99100
arch: "amd64"
100101
container_image: "rapidsai/ci-conda:latest"
101-
run_script: "ci/test_notebooks.sh"
102+
script: "ci/test_notebooks.sh"
102103
wheel-tests-cudf:
103104
secrets: inherit
104105
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.06
@@ -136,7 +137,7 @@ jobs:
136137
sha: ${{ inputs.sha }}
137138
node_type: "gpu-l4-latest-1"
138139
container_image: "rapidsai/ci-conda:cuda12.9.0-ubuntu24.04-py3.12"
139-
run_script: |
140+
script: |
140141
ci/cudf_pandas_scripts/third-party-integration/test.sh python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml
141142
wheel-tests-cudf-polars:
142143
secrets: inherit
@@ -166,4 +167,4 @@ jobs:
166167
sha: ${{ inputs.sha }}
167168
node_type: "gpu-l4-latest-1"
168169
container_image: "rapidsai/ci-conda:latest"
169-
run_script: ci/test_narwhals.sh
170+
script: ci/test_narwhals.sh

‎ci/build_wheel.sh

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ package_dir=$2
88

99
source rapids-configure-sccache
1010
source rapids-date-string
11+
source rapids-init-pip
1112

1213
rapids-generate-version > ./VERSION
1314
rapids-generate-version > ./python/cudf/cudf/VERSION

‎ci/build_wheel_cudf.sh

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,20 +3,21 @@
33

44
set -euo pipefail
55

6+
source rapids-init-pip
7+
68
package_dir="python/cudf"
79

810
RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen "${RAPIDS_CUDA_VERSION}")"
911

1012
# Downloads libcudf and pylibcudf wheels from this current build,
1113
# then ensures 'cudf' wheel builds always use the 'libcudf' and 'pylibcudf' just built in the same CI run.
1214
#
13-
# Using env variable PIP_CONSTRAINT is necessary to ensure the constraints
15+
# Using env variable PIP_CONSTRAINT (initialized by 'rapids-init-pip') is necessary to ensure the constraints
1416
# are used when creating the isolated build environment.
1517
LIBCUDF_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="libcudf_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-github cpp)
1618
PYLIBCUDF_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="pylibcudf_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-github python)
17-
echo "libcudf-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${LIBCUDF_WHEELHOUSE}/libcudf_*.whl)" > /tmp/constraints.txt
18-
echo "pylibcudf-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${PYLIBCUDF_WHEELHOUSE}/pylibcudf_*.whl)" >> /tmp/constraints.txt
19-
export PIP_CONSTRAINT="/tmp/constraints.txt"
19+
echo "libcudf-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${LIBCUDF_WHEELHOUSE}/libcudf_*.whl)" >> "${PIP_CONSTRAINT}"
20+
echo "pylibcudf-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${PYLIBCUDF_WHEELHOUSE}/pylibcudf_*.whl)" >> "${PIP_CONSTRAINT}"
2021

2122
./ci/build_wheel.sh cudf ${package_dir}
2223

‎ci/build_wheel_cudf_polars.sh

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@
33

44
set -euo pipefail
55

6+
source rapids-init-pip
7+
68
package_dir="python/cudf_polars"
79

810
./ci/build_wheel.sh cudf-polars ${package_dir}

‎ci/build_wheel_dask_cudf.sh

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@
33

44
set -euo pipefail
55

6+
source rapids-init-pip
7+
68
package_dir="python/dask_cudf"
79

810
./ci/build_wheel.sh dask-cudf ${package_dir}

‎ci/build_wheel_libcudf.sh

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@
33

44
set -euo pipefail
55

6+
source rapids-init-pip
7+
68
package_name="libcudf"
79
package_dir="python/libcudf"
810

‎ci/build_wheel_pylibcudf.sh

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,18 +3,19 @@
33

44
set -euo pipefail
55

6+
source rapids-init-pip
7+
68
package_dir="python/pylibcudf"
79

810
RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen "${RAPIDS_CUDA_VERSION}")"
911

1012
# Downloads libcudf wheel from this current build,
1113
# then ensures 'pylibcudf' wheel builds always use the 'libcudf' just built in the same CI run.
1214
#
13-
# Using env variable PIP_CONSTRAINT is necessary to ensure the constraints
15+
# Using env variable PIP_CONSTRAINT (initialized by 'rapids-init-pip') is necessary to ensure the constraints
1416
# are used when creating the isolated build environment.
1517
LIBCUDF_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="libcudf_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-github cpp)
16-
echo "libcudf-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${LIBCUDF_WHEELHOUSE}/libcudf_*.whl)" > /tmp/constraints.txt
17-
export PIP_CONSTRAINT="/tmp/constraints.txt"
18+
echo "libcudf-${RAPIDS_PY_CUDA_SUFFIX} @ file://$(echo ${LIBCUDF_WHEELHOUSE}/libcudf_*.whl)" >> "${PIP_CONSTRAINT}"
1819

1920
./ci/build_wheel.sh pylibcudf ${package_dir}
2021

‎ci/cudf_pandas_scripts/pandas-tests/run.sh

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@
55

66
set -euo pipefail
77

8+
source rapids-init-pip
9+
810
EXITCODE=0
911
trap "EXITCODE=1" ERR
1012
set +e

0 commit comments

Comments
 (0)