How much memory is needed for the Nightly Build Docker?

[bdbowyer]

I am having issues with the addmm nightly test in Tritonbench on my own Docker, so I tried running with the Nightly Docker:
https://github.com/pytorch-labs/tritonbench/blob/main/docker/tritonbench-nightly.dockerfile

I have a time out after an hour on a build machine with 16GB of RAM. How much RAM is needed to run the nightly build? What is the expected build time?

#30 627.1 Compiling src/device/onerank.cu

#30 627.1 nvcc warning : Support for offline compilation for architectures prior to '<compute/sm/lto>_75' will be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).

Build timed out (retrying in 2s)
Configuring buildkit
Using buildkit builder pod: gcp-builder-v0.11.2-5dc7797fbf-c9lxt
Checking builder pod health
Builder pod healthy
Logging into Docker registries
Build errored for triton-bench!
Build timed out

[xuzhao9]

We run the nightly docker build at https://github.com/pytorch-labs/tritonbench/actions/workflows/docker.yaml, and it takes ~3h to complete the build.

A large chunk of the build time is for building flash_attn and fbgemm_gpu packages. If you are not interested in testing these, the build time and memory requirements can be largely reduced.

In the CI, we use Meta-managed 32-core-ubuntu type CPU-only GHA runners to build the nightly docker. I haven't tested, but I think it should be 128 GB memory.