[RELEASE] rapidsmpf v25.06 #322
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Authors: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Jake Awe (https://github.com/AyodeAwe) URL: #159
We decided that since CUDA 11 support is being targeted to be dropped from RAPIDS around August there's no point for us to support it only for a few more months and thus are dropping it immediately. Authors: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Gil Forsyth (https://github.com/gforsyth) URL: #144
The nightly GHA workflow has been failing when calling the build workflow, this is happening because `docs-build` refers `python-build` which is an inexistent job, the correct name is `conda-python-build` which this PR is now fixing. Authors: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Gil Forsyth (https://github.com/gforsyth) URL: #160
Authors: - Mads R. B. Kristensen (https://github.com/madsbk) - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) URL: #146
This removes numpydoc validation from our pre-commit config. We'll rely just on the docstring validation provided by pre-commit. #156 (comment) has some background on the issue. #156 (comment) has an overview of what we lose. Authors: - Tom Augspurger (https://github.com/TomAugspurger) - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Jake Awe (https://github.com/AyodeAwe) - Mads R. B. Kristensen (https://github.com/madsbk) URL: #156
This should provide more confidence in code quality and help prevent bugs on issues like integer overflow. Authors: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #158
Fix build errors that were introduced because #146 was merged before #158 , without the latter having updated the branch. Authors: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Niranda Perera (https://github.com/nirandaperera) URL: #163
@nirandaperera noticed that I forget to register a spill function in #146. Fixed in this PR, which also includes some more tests. Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Niranda Perera (https://github.com/nirandaperera) URL: #165
Add a `ProgressThread` built on top of `PausableThreadLoop` that allows registering and progressing generic functions. This PR only adds the `ProgressThread` but does not use it anywhere, that part will be done in a follow-up PR. Authors: - Peter Andreas Entschev (https://github.com/pentschev) - Niranda Perera (https://github.com/nirandaperera) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Niranda Perera (https://github.com/nirandaperera) URL: #143
This closes #161 Hopefully this will give us more information regarding #152 flaky tests Authors: - Niranda Perera (https://github.com/nirandaperera) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #164
This refactors how we do our dask cluster boostraping. It fixes a few issues: 1. No need for a new Cluster class. Just use LocalCUDACluster. 2. Avoid using `Client.submit` and `Futures` to bootstrap the UCXX comms, which show up in the user's Dashboard. 3. Avoid a "two-stage" bootstrapping, where we have an initial stage that sets up the root node followed by bootstrapping the other nodes. Worker plugins are applied uniformly to each worker, so the plugin itself needs to have a way to handle both the root and non-root node cases. Instead, we'll use scheduler and worker plugins. These are all set up in `bootstrap_dask_cluster`, which is called via `rapidsmp.integrations.dask.get_client` (which is called via `rapidsmp_shuffle_graph)`. Users shouldn't need to change anything. The actual implementation grew a little complicated, but I've [left a note](https://github.com/rapidsai/rapids-multi-gpu/pull/150/files#diff-79c8049ca8ea9f7d1c3f247e46853db9eb5e9c68647ac49e7ebb5aca7dc7c312R763) explaining things. That might be a good spot to start reviewing. Authors: - Tom Augspurger (https://github.com/TomAugspurger) Approvers: - Richard (Rick) Zamora (https://github.com/rjzamora) - Peter Andreas Entschev (https://github.com/pentschev) URL: #150
Closes #168 Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) URL: #170
We emit a barrier in the TearDown of every test to ensure there's no overlap in messages between different test instances. However, we need to do that on the communicator we're actually using in the test suite. Authors: - Lawrence Mitchell (https://github.com/wence-) - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Niranda Perera (https://github.com/nirandaperera) - Peter Andreas Entschev (https://github.com/pentschev) URL: #171
This is a follow-up to #150 Closes #157 The goal of this PR is to simplify memory-resource creation by avoiding it in `rapidsmp` altogether. Since the user can just use `LocalCUDACluster` to deploy a Dask cluster, they can also use existing options/utilities to create a memory pool on each worker. When `rapidsmp.integrations.dask.bootstrap_dask_cluster` is called, each worker only needs to wrap the current memory resource in a `StatisticsResourceAdaptor`. This is technically "breaking", because it removes the `pool_size` argument from `bootstrap_dask_cluster`. However, we are only using that option in rapidsai/cudf#18335 (which is still experimental - and can be easily changed). Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - Tom Augspurger (https://github.com/TomAugspurger) - Peter Andreas Entschev (https://github.com/pentschev) URL: #172
Authors: - Niranda Perera (https://github.com/nirandaperera) - Ayush Dattagupta (https://github.com/ayushdg) - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) - Mads R. B. Kristensen (https://github.com/madsbk) - Ayush Dattagupta (https://github.com/ayushdg) - Lawrence Mitchell (https://github.com/wence-) - Bradley Dice (https://github.com/bdice) URL: #105
Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) - Lawrence Mitchell (https://github.com/wence-) URL: #169
Ensure `SpillOnInsertAndExtraction` is tested with UCXX as well. To do that a new method `split()` is added to `GlobalEnvironment` so that both MPI and UCXX communicators can be split, since this functionality didn't exist for the UCXX communicator a new method was added to perform that task, similar to how `barrier()` was already implemented that mimics MPI. Authors: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #167
Create `integration/dask/` and split up the dask-core from the shuffler integration. This PR is only reorganization. This is motivated by my work on dask-spilling integration, which will also need its own submodule. Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) URL: #174
Follow-up to #174 Allows us to avoid changes in `cudf_polars` (as mentioned in #174 (comment)). If we don't want to add these imports to `rapidsmp/integrations/dask/__init__.py`, we just need to update the import paths in `cudf_polars`. Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #176
Pushing this up a bit early for @nirandaperera to take a look. Will add more details later. Closes #125 Authors: - Tom Augspurger (https://github.com/TomAugspurger) - Niranda Perera (https://github.com/nirandaperera) - Peter Andreas Entschev (https://github.com/pentschev) - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Lawrence Mitchell (https://github.com/wence-) URL: #130
Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Tom Augspurger (https://github.com/TomAugspurger) - Peter Andreas Entschev (https://github.com/pentschev) URL: #177
The core of the shuffle algorithms just operates on bytes, rather than being specifically tied to libcudf objects. To start separating these, introduce our own "packed" data format that has bag-of-bytes semantics with metadata. Authors: - Lawrence Mitchell (https://github.com/wence-) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) URL: #179
Use `ProgressThread` in `Shuffler` so that all instances of `Shuffler` may share the same thread since the work in the progress thread generally requires really low compute capacity, as it's only progressing asynchronous tasks whose heavy-lifting is done on other threads. However, it's still easy to add use multiple threads if so chosen, then user can simply instantiate multiple `ProgressThread`s for each `Shuffler` or groups of `Shuffler`s. Authors: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Niranda Perera (https://github.com/nirandaperera) URL: #166
Authors: - Niranda Perera (https://github.com/nirandaperera) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Lawrence Mitchell (https://github.com/wence-) URL: #178
Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) - Lawrence Mitchell (https://github.com/wence-) URL: #182
Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) URL: #184
Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) URL: #185
It's very important to release the GIL before calling into any C++ API that might block (e.g., acquiring a lock, waiting for communication, etc.). Otherwise, the thread could block while still holding the GIL, potentially leading to a deadlock. Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) URL: #188
Use `Duration` everywhere Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) URL: #189
Include `cmake/RAPIDS.cmake` so it doesn't need to be downloaded anymore, avoiding hitting rate-limits. Authors: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Bradley Dice (https://github.com/bdice) - Mads R. B. Kristensen (https://github.com/madsbk) URL: #183
Contributes to rapidsai/build-planning#120 This PR adds support for Python 3.13. ## Notes for Reviewers This is part of ongoing work to add Python 3.13 support across RAPIDS. It temporarily introduces a build/test matrix including Python 3.13, from rapidsai/shared-workflows#268. A follow-up PR will revert back to pointing at the `branch-25.06` branch of `shared-workflows` once all RAPIDS repos have added Python 3.13 support. ### This will fail until all dependencies have been updated to Python 3.13 CI here is expected to fail until all of this project's upstream dependencies support Python 3.13. This can be merged whenever all CI jobs are passing. Authors: - Gil Forsyth (https://github.com/gforsyth) Approvers: - Bradley Dice (https://github.com/bdice) URL: #262
This PR uses CUDA 12.9.0 to build and test. xref: rapidsai/build-planning#173 Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Gil Forsyth (https://github.com/gforsyth) URL: #264
Follow up to #249 - Adds minimal changes to support sorting with a `DaskIntegration` protocol. - Does **not** bother to implement/demonstrate anything beyond an ascending sort by a single column. - Adds `sort_boundaries` and `options` arguments to `DaskIntegration.insert_partition` and adds an `options` argument to `DaskIntegration.extract_partition` - The `sort_boundaries` must be a separate positional argument so that it can be generated dynamically as the output of a separate task. All other sorting options can be passed through with `options` (e.g. ascending vs descending and null precedence). This technically breaks existing cudf-polars code, so we will want to update the `DaskIntegration` protocol defined in cudf asap after this is finished/merged. Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - Sebastian Berg (https://github.com/seberg) - Tom Augspurger (https://github.com/TomAugspurger) URL: #256
Currently, the `librapidsmpf` packages ship gtest/gmock library and include files, which they shouldn't. Simplifying this down to the current version also in libcudf should fix that (i.e. by linking statically). I would just do this to keep similar to stay similar to other rapids libraries, unless this is somehow different? ~(I think there is another small issue with `mpi_tests` and `ucxx_tests` getting shipped in `librapidsmpf` rather than `librapidsmpf-tests`, but I'll have a look at it seperately.)~ Authors: - Sebastian Berg (https://github.com/seberg) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Peter Andreas Entschev (https://github.com/pentschev) - Gil Forsyth (https://github.com/gforsyth) URL: #261
Closes #219 Locally, this works for me in a no-mpi conda env: ``` ./build.sh -v -g --cmake-args=\"-DBUILD_MPI_SUPPORT=OFF -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DBUILD_EXAMPLES=OFF\" librapidsmpf rapidsmpf && pytest python/rapidsmpf/rapidsmpf/tests/test_dask.py -vsx ``` Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) - Lawrence Mitchell (https://github.com/wence-) URL: #260
Ray packages for Python 3.13 are not yet available in conda-forge, thus skip testing on Python 3.13 for now. Authors: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Bradley Dice (https://github.com/bdice) URL: #272
Contributes to rapidsai/build-infra#237 Switches package downloads in CI away from `downloads.rapids.ai`, to GitHub Actions artifact store. This is part of an ongoing migration across RAPIDS to use the GitHub Actions artifact store. ## Notes for Reviewers I found these like this: ```shell git grep -E '\-s3' git grep -i downloads ``` Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) - Bradley Dice (https://github.com/bdice) URL: #274
AFAICT spdlog is entirely unused. Best to remove it before it accidentally gets used and causes us the same old headaches. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) URL: #277
Current PackedData cython API does not contain a constructor that takes in cudf packed columns. This PR adds it. Authors: - Niranda Perera (https://github.com/nirandaperera) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) - Mads R. B. Kristensen (https://github.com/madsbk) URL: #228
This PR reverts shared-workflows branches to `branch-25.06` and ensures builds use CUDA 12.8. CUDA 12.9 will be used in the test matrix but not for builds in RAPIDS 25.06. See rapidsai/build-planning#173 (comment) for more information. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Gil Forsyth (https://github.com/gforsyth) URL: #282
Authors: - Tom Augspurger (https://github.com/TomAugspurger) Approvers: - Richard (Rick) Zamora (https://github.com/rjzamora) URL: #284
RAPIDS' build-docs default has switched to a Python 3.13 environment. However, Ray for Python 3.13 is not available in conda-forge yet, thus we allow installing from PyPI for the time being just to ensure docs can build successfully, even though it may not be usable if installed on from PyPI on a conda environment. See also #272. Authors: - Peter Andreas Entschev (https://github.com/pentschev) Approvers: - Bradley Dice (https://github.com/bdice) URL: #286
This PR adds the following utils to the `Buffer` class. These will be used in the Chunk in a follow up PR.
```c++
public:
/**
* @brief Copy data from this buffer to a destination buffer with a given offset.
*
* @param dest Destination buffer.
* @param dest_offset Offset of the destination buffer.
* @param stream CUDA stream to use for the copy.
* @returns Number of bytes written to the destination buffer.
* @throws std::invalid_argument if copy violates the bounds of the destination
* buffer.
*/
[[nodiscard]] std::ptrdiff_t copy_to(
Buffer& dest, std::ptrdiff_t dest_offset, rmm::cuda_stream_view stream
) const;
private:
/**
* @brief Copy a slice of the buffer to a new buffer.
*
* @param offset Offset in bytes from the start of the buffer.
* @param length Length in bytes of the slice.
* @param stream CUDA stream to use for the copy.
* @returns A new buffer containing the copied slice.
*/
[[nodiscard]] std::unique_ptr<Buffer> copy_slice(
std::ptrdiff_t offset, std::ptrdiff_t length, rmm::cuda_stream_view stream
) const;
/**
* @brief Copy a slice of the buffer to a new buffer.
*
* @param target Memory type of the new buffer.
* @param offset Offset in bytes from the start of the buffer.
* @param length Length in bytes of the slice.
* @param stream CUDA stream to use for the copy.
* @returns A new buffer containing the copied slice.
*/
[[nodiscard]] std::unique_ptr<Buffer> copy_slice(
MemoryType target,
std::ptrdiff_t offset,
std::ptrdiff_t length,
rmm::cuda_stream_view stream
) const;
```
Authors:
- Niranda Perera (https://github.com/nirandaperera)
Approvers:
- Mads R. B. Kristensen (https://github.com/madsbk)
- Lawrence Mitchell (https://github.com/wence-)
- Peter Andreas Entschev (https://github.com/pentschev)
URL: #270
Implement out-of-memory protection by using a RMM resource `RmmFallbackResource` based on rapidsai/rmm#1665. The idea is to use managed memory when the RMM pool raises an OOM error. Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) URL: #287
This PR lays the groundwork for a centralized configuration system in RapidsMPF, allowing options to be passed through the stack without relying on global dictionaries or modifying function signatures.
Currently, only the logging level option is implemented as a proof of concept. Additional options will be introduced in future PRs.
#### Key Features
- Works seamlessly in both Python and C++.
- Efficient: options are initialized only once.
- Supports integration with environment variables while maintaining explicit control — important because environment variable access is not thread-safe.
#### C++ Snippet
```c++
// Initialize configuration options from environment variables.
rapidsmpf::config::Options options{rapidsmpf::config::get_environment_variables()};
// Pass it to the communicator
std::shared_ptr<rapidsmpf::Communicator> comm =
std::make_shared<rapidsmpf::MPI>(MPI_COMM_WORLD, options);
...
// In its ctor, the communicator can then use `options.get` to retrieve the `LOG_LEVEL` enum.
// `level_from_string` is called once to convert the string representation in `options` to a `LOG_LEVEL`.
Communicator::Logger::Logger(Communicator* comm, config::Options options)
: comm_{comm}, level_(options.get<LOG_LEVEL>("log", level_from_string)){};
```
Authors:
- Mads R. B. Kristensen (https://github.com/madsbk)
Approvers:
- Peter Andreas Entschev (https://github.com/pentschev)
URL: #285
Followup to #208, we need to do this everywhere. Authors: - Lawrence Mitchell (https://github.com/wence-) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Peter Andreas Entschev (https://github.com/pentschev) URL: #294
Removing `OptionValue`'s ctor that takes `std::any`. Initially, I used SFINAE to fix GCC version 11's ctor confusion but I think it is better to always require a string representation. Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Lawrence Mitchell (https://github.com/wence-) - Peter Andreas Entschev (https://github.com/pentschev) URL: #297
Proposes a batch of small, mostly-unrelated changes to building and packaging. These are all part of RAPIDS-wide efforts to make it easier to reproduce CI locally and to use artifacts from one project's CI in another project's CI. Contributes to rapidsai/shared-workflows#356 * explicitly provides an input for `script` to workflows using it, instead of relying on a default value coming from the workflow file Contributes to rapidsai/shared-workflows#337 * switches from input `run_script` to `script` in uses of the `custom-job` workflow ## Notes for Reviewers # Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #293
Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) URL: #299
Ports the conda recipes over to use `rattler-build` instead of `conda-build` Authors: - Gil Forsyth (https://github.com/gforsyth) Approvers: - James Lamb (https://github.com/jameslamb) URL: #289
#309 made this same change, but against `branch-25.08`... this configuration needs to be updated on the default branch, which is currently `branch-25.06`.
Removing `Options.get_or_assign()`, which is now replaced by the more generic `Options.get()`. Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Tom Augspurger (https://github.com/TomAugspurger) - Bradley Dice (https://github.com/bdice) URL: #303
This PR adds the google benchmarking suite to the build and a benchmark for PatitionAndPack operation. Bench [results](https://docs.google.com/spreadsheets/d/1eKD-6zcbjgIBwAq8YIUE1hAGoqFPu4gT-K8T-kW_Dv0/edit?gid=0#gid=0) Authors: - Niranda Perera (https://github.com/nirandaperera) Approvers: - Bradley Dice (https://github.com/bdice) - Tom Augspurger (https://github.com/TomAugspurger) URL: #307
xref #219 Part of rapidsai/build-planning#170 Adds wheels for `rapidsmpf`. Currently the `librapidsmpf` and `rapidsmpf` wheels are combined into a single file. My CMake isn't great, and for now I'm hacking in some paths with an environment variable, just so I can start testing this in CI, but I'll work on something more idiomatic. Authors: - Gil Forsyth (https://github.com/gforsyth) - Mads R. B. Kristensen (https://github.com/madsbk) - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Bradley Dice (https://github.com/bdice) - Tom Augspurger (https://github.com/TomAugspurger) - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #269
This PR adds a separate arg for output partition count for the shuffle benchmark. Previously, output partition count was derived as `input_partition_count*nranks`. When running weak scaling experiments, this introduces a quadratic component to the `PartitionAndPack` loop. This muddles the timing values, indicating that shuffler scales really poorly on multiple partitions (1x4GB partition vs 64x64MB partitions). In reality, this is due to the increased number of calls to `PartitionAndPack`. Corrected [results](http://docs.google.com/document/d/1hRojt0w4UOVH8Ga4J7qKMpuZZ2rj0JAQQ9uFA7JB2k4/edit?tab=t.0#heading=h.w2vyoagnu7v2) Authors: - Niranda Perera (https://github.com/nirandaperera) Approvers: - Tom Augspurger (https://github.com/TomAugspurger) URL: #316
Followup to #269 Adds wheel publishing to https://pypi.anaconda.org/rapidsai-wheels-nightly/simple/ on merges to `branch-25.06` + nightly runs. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Gil Forsyth (https://github.com/gforsyth) URL: #319
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
❄️ Code freeze for
branch-25.06and v25.06 releaseWhat does this mean?
Only critical/hotfix level issues should be merged into
branch-25.06until release (merging of this PR).What is the purpose of this PR?
branch-25.06intomainfor the release