feat(torch): Update torch libraries to v2.5.0, bundle triton, patch TransformerEngine
#85
Conversation
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368570 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368570 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368570 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368570 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368570 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368567 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368570 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368567 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368567 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368567 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368567 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368567 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368570 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368570 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368570 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368570 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368567 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368570 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368570 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368567 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368567 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368567 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368567 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368567 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
|
@Eta0 Build complete, success: https://github.com/coreweave/ml-containers/actions/runs/11469368645 |
PyTorch v2.5.0,
triton, & Patched TransformerEngine v1.11This change includes version updates, some patches, and bundles a source build of
tritonin the non-nightlyml-containers/torchimages (as they were previously only bundled in the nightly ones).PyTorch
The following PyTorch components have been updated:
torchv2.4.1 → v2.5.0torchvisionv0.19.1 → v0.20.0torchaudiov2.4.1 → v2.5.0In addition, the patch to fix
torchaudiocompilation on CUDA 12.5+ (from #82) is now obsolete in the nightly builds oftorchaudio(so, for some version after v2.5.0), as the contents of the patch were included as a secondary change in pytorch/audio#3843, so this adds a check to the build process to only apply that patch in versions where it is necessary.tritontritonis normally listed as a dependency for the x86_64 Linux releases of PyTorch on PyPI, but it is not automatically registered as a dependency in our images given the way that we build PyTorch from source. Because it would normally be seen as a required dependency, and because each PyTorch release expects a specific version oftritonto be paired with it (when using features that requiretriton), this PR adds a source build oftritonas a bundled part of allml-containers/torchimages.The version of
tritonto build is pulled with the same method thattorch-nightlyimages use: a commit hash is read from the.ci/docker/ci_commit_pins/triton.txtfile in the clonedpytorch/pytorchrepository, and that exact git commit oftritonis used for a build. PyPI releases oftritonare not used, even though they are generally compatible with stable PyTorch versions built from source, like ours, as we already had the code to buildtritonfrom a specific commit, and this could potentially be more flexible when building e.g. PyTorch release candidates.The version of
tritonto be built can still be overridden by setting theBUILD_TRITON_VERSIONbuild argument explicitly, and can be turned off completely by setting the new separateBUILD_TRITONbuild argument to any value other than1.TransformerEngine Updates
This change additionally updates TransformerEngine by a few commits, as it was previously using a commit from right before the v1.11 release. Now that v1.11 is out, it builds from that tag, and additionally includes a patch for a bug in the v1.11 release (NVIDIA/TransformerEngine#1213) that was fixed on the v1.11 branch (and in future versions) but not the v1.11 git tag as part of NVIDIA/TransformerEngine#1222.