Why does vLLM use a custom all-reduce method?

[SamKG]

Hello,
Was hoping someone could help shed some light on this -
why does vLLM choose to use a custom all-reduce method? Is there a benefit to doing this over just using the NCCL APIs?

[simon-mo]

See perf result here #2192. In certain cases, the custom topology drastically boosts performance compared to nccl's implementation. vLLM still uses nccl in majority of cases.

[SamKG]

thanks, that helps a ton!!

[SamKG]

@simon-mo Just wanted to quickly clarify something: I've observed strange scaling in performance for the 1-stage all-reduce kernels (both nccl and for vllm):

For small problem sizes, it actually performs slower than expected (this is the region where 1-stage reduce is being used)

Theoretically, we should expect scalings that look (roughly) like the below:

Is this expected?

[SamKG]

cc @hanzhi713

[cduk]

In a set-up where 4 GPUs are connected by PCIe, but each pair of GPUs are connected by NVLink (112 GB/s bi-directional). Is there a way to specify a reduction first on each pairwise bound set of GPUs before reducing across the slower PCIe link?