Skip to content

feat: Nvidia GPU Direct Storage Support for reading RNTuple #1426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 30 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
6ab3dc5
Add nvidia GDS support for RNTuple reading.
fstrug Apr 10, 2025
e1e2184
Merge branch 'scikit-hep:main' into main
fstrug Apr 16, 2025
dc63aa2
Merge branch 'scikit-hep:main' into main
fstrug Apr 22, 2025
1daaeb4
Integrate RNTuple GDS functionallity across behaviors/RNTuple.py and …
fstrug Apr 22, 2025
e81912c
style: pre-commit fixes
pre-commit-ci[bot] Apr 22, 2025
8b030a1
Add support for LZ4 decompression. Update some RNTuple tests to verif…
fstrug May 6, 2025
55ff9f4
Resolve merge conflicts
fstrug May 6, 2025
3c4dd08
Parametetrized more RNTuple pytests. Added support for reading surpre…
fstrug May 14, 2025
ceac95d
Remove unnecessary imports. GDS dependencies imported through . Updat…
fstrug May 14, 2025
84c7929
Merge branch 'main' into main
fstrug May 14, 2025
bcf6a05
style: pre-commit fixes
pre-commit-ci[bot] May 14, 2025
6eda90a
More tests updated.
fstrug May 14, 2025
0d206c7
style: pre-commit fixes
pre-commit-ci[bot] May 14, 2025
a1c5fa6
Stashing changes
fstrug May 29, 2025
55b7925
style: pre-commit fixes
pre-commit-ci[bot] May 29, 2025
546195e
Fixed bug causing repeated deserialization operations on a column of …
fstrug May 29, 2025
eb15117
Added initial implementation for interface to kvikio.CuFile.
fstrug May 29, 2025
333397d
style: pre-commit fixes
pre-commit-ci[bot] May 29, 2025
dd02c45
Merge pull request #1 from scikit-hep/main
fstrug May 29, 2025
8fb1921
style: pre-commit fixes
pre-commit-ci[bot] May 29, 2025
0d9e970
Added doc strings. Code cleanup.
fstrug May 30, 2025
41755b3
style: pre-commit fixes
pre-commit-ci[bot] May 30, 2025
3fd629f
Fixed linter bugs
fstrug Jun 10, 2025
7fc0520
Merge pull request #2 from scikit-hep/main
fstrug Jun 10, 2025
28e6ae2
Skip GDS tests if no available CUDA driver.
fstrug Jun 10, 2025
ac31628
GDS tests should only run on supported OS with available cuda driver.
fstrug Jun 11, 2025
753ac69
Linting
fstrug Jun 11, 2025
4bc0c94
Update pyproject.toml
fstrug Jun 12, 2025
e6c0163
Update pyproject.toml
fstrug Jun 12, 2025
4b3c46e
Update pyproject.toml
fstrug Jun 12, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,20 @@ readme = "README.md"
requires-python = ">=3.9"

[project.optional-dependencies]
GDS_cu11 = [
"kvikio-cu11>=25.02.01"
]
GDS_cu12 = [
"kvikio-cu12>=25.02.01"
]
dev = [
"boost_histogram>=0.13",
"dask-awkward>=2025.2.0",
"dask[array,distributed]",
"hist>=1.2",
"pandas",
"awkward-pandas"
]
http = ["aiohttp"]
s3 = ["s3fs"]
xrootd = ["fsspec-xrootd>=0.5.0"]
Expand Down
371 changes: 368 additions & 3 deletions src/uproot/behaviors/RNTuple.py

Large diffs are not rendered by default.

48 changes: 48 additions & 0 deletions src/uproot/extras.py
Original file line number Diff line number Diff line change
Expand Up @@ -345,3 +345,51 @@ def awkward_pandas():
) from err
else:
return awkward_pandas


def cupy():
"""
Imports and returns ``cupy``.
"""
try:
import cupy
except ModuleNotFoundError as err:
raise ModuleNotFoundError(
"""Cupy is required for GDS reading to work. Please install GDS dependencies with:
`python3 -m pip install uproot[GDS_cuX]`
where X is the cuda major version on user's system (11 and 12 currently supported). Cuda major version can be checked by calling `nvidia-smi --version` or `nvcc --version` if available."""
) from err
else:
return cupy


def kvikio():
"""
Imports and returns ``kvikio``.
"""
try:
import kvikio
except ModuleNotFoundError as err:
raise ModuleNotFoundError(
"""Kvikio is required for GDS reading to work. Please install GDS dependencies with:
`python3 -m pip install uproot[GDS_cuX]`
where X is the cuda major version on user's system (11 and 12 currently supported). Cuda major version can be checked by calling `nvidia-smi --version` or `nvcc --version` if available."""
) from err
else:
return kvikio


def kvikio_nvcomp_codec():
"""
Imports and returns ``kvikio``.
"""
try:
import kvikio.nvcomp_codec as kvikio_nvcomp_codec
except ModuleNotFoundError as err:
raise ModuleNotFoundError(
"""Kvikio is required for GDS reading to work. Please install GDS dependencies with:
`python3 -m pip install uproot[GDS_cuX]`
where X is the cuda major version on user's system (11 and 12 currently supported). Cuda major version can be checked by calling `nvidia-smi --version` or `nvcc --version` if available."""
) from err
else:
return kvikio_nvcomp_codec
Loading
Loading