Potential Hardware Failure when running vllm

[NicolasDrapier]

Description:

I've noticed some strange behavior when using vllm. After running a few requests on the server, my hardware sometimes crashes completely—this includes all four physical power supplies shutting off.

I followed the hardware testing procedures as outlined in this guide, and all tests passed successfully.

# docker exec 37b808c71bf7 /usr/bin/dcgmi diag --run 4 --fail-early
Successfully ran diagnostic for group.

+--------------------------------+-----------------------------------------+
|           Diagnostic           |     Result                              |
+--------------------------------+-----------------------------------------+
|           Metadata             |                                         |
| DCGM Version                   | 3.3.7                                   |
| Driver Version Detected        | 550.67                                  |
| GPU Device IDs Detected        | 26b9,26b9,26b9,26b9,26b9,26b9,26b9,26b9 |
+--------------------------------+-----------------------------------------+
|           Deployment           |                                         |
| DenyList                       | Pass                                    |
| NVML Library                   | Pass                                    |
| CUDA Main Library              | Pass                                    |
| Permissions and OS Blocks      | Pass                                    |
| Persistence Mode               | Pass                                    |
| Environment Variables          | Pass                                    |
| Page Retirement/Row Remap      | Pass                                    |
| Graphics Processes             | Pass                                    |
| Inforom                        | Pass                                    |
+--------------------------------+-----------------------------------------+
|           Integration          |                                         |
| PCIe                           | Pass - All                              |
+--------------------------------+-----------------------------------------+
|           Hardware             |                                         |
| GPU Memory                     | Pass - All                              |
| Diagnostic                     | Pass - All                              |
| Pulse Test                     | Pass - All                              |
+--------------------------------+-----------------------------------------+
|           Stress               |                                         |
| Targeted Stress                | Pass - All                              |
| Targeted Power                 | Pass - All                              |
| Memory Bandwidth               | Pass - All                              |
| Memtest                        | Pass - All                              |
| EUV Test                       | Skip - All                              |
+--------------------------------+-----------------------------------------+

However, there are instances where, if all four power supplies don't shut off, one or two of them will physically turn off. I tried running the same workload with text-generation-inference, and while I did not encounter a full server crash, some of the power supplies still tripped.

My Questions:

• Could there be potential issues with the compiled CUDA kernels that might cause this type of behavior?
• Has anyone else encountered this problem?
• If not, what steps should I follow to debug this issue?

Setup:

PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: openSUSE Tumbleweed (x86_64)
GCC version: (SUSE Linux) 13.2.1 20240206 [revision 67ac78caf31f7cb3202177e6428a46d829b70f23]
Clang version: Could not collect
CMake version: version 3.30.1
Libc version: glibc-2.39

Python version: 3.11.9 (main, Apr 08 2024, 06:18:15) [GCC] (64-bit runtime)
Python platform: Linux-6.8.5-1-default-x86_64-with-glibc2.39
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA L40S
GPU 1: NVIDIA L40S
GPU 2: NVIDIA L40S
GPU 3: NVIDIA L40S
GPU 4: NVIDIA L40S
GPU 5: NVIDIA L40S
GPU 6: NVIDIA L40S
GPU 7: NVIDIA L40S

Nvidia driver version: 550.67
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        52 bits physical, 57 bits virtual
Byte Order:                           Little Endian
CPU(s):                               48
On-line CPU(s) list:                  0-47
Vendor ID:                            AuthenticAMD
BIOS Vendor ID:                       Advanced Micro Devices, Inc.
Model name:                           AMD EPYC 9254 24-Core Processor
BIOS Model name:                      AMD EPYC 9254 24-Core Processor                 Unknown CPU @ 2.9GHz
BIOS CPU family:                      107
CPU family:                           25
Model:                                17
Thread(s) per core:                   1
Core(s) per socket:                   24
Socket(s):                            2
Stepping:                             1
Frequency boost:                      enabled
CPU(s) scaling MHz:                   40%
CPU max MHz:                          4151.7568
CPU min MHz:                          1500.0000
BogoMIPS:                             5793.30
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap
Virtualization:                       AMD-V
L1d cache:                            1.5 MiB (48 instances)
L1i cache:                            1.5 MiB (48 instances)
L2 cache:                             48 MiB (48 instances)
L3 cache:                             256 MiB (8 instances)
NUMA node(s):                         2
NUMA node0 CPU(s):                    0-23
NUMA node1 CPU(s):                    24-47
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Mitigation; Safe RET
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] flashinfer==0.1.1+cu121torch2.3
[pip3] mypy-extensions==1.0.0
[pip3] mypy-protobuf==3.6.0
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.535.133
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.6.20
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pytorch-ranger==0.1.1
[pip3] pyzmq==26.0.0
[pip3] torch==2.4.0
[pip3] torch-optimizer==0.3.0
[pip3] torchaudio==2.4.0
[pip3] torchmetrics==1.3.2
[pip3] torchvision==0.19.0
[pip3] transformers==4.43.1
[pip3] triton==3.0.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.4@4db5176d9758b720b05460c50ace3c01026eb158
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PIX     SYS     SYS     SYS     SYS     SYS     SYS     0-23    0               N/A
GPU1    PIX      X      SYS     SYS     SYS     SYS     SYS     SYS     0-23    0               N/A
GPU2    SYS     SYS      X      PIX     SYS     SYS     SYS     SYS     0-23    0               N/A
GPU3    SYS     SYS     PIX      X      SYS     SYS     SYS     SYS     0-23    0               N/A
GPU4    SYS     SYS     SYS     SYS      X      PIX     SYS     SYS     24-47   1               N/A
GPU5    SYS     SYS     SYS     SYS     PIX      X      SYS     SYS     24-47   1               N/A
GPU6    SYS     SYS     SYS     SYS     SYS     SYS      X      PIX     24-47   1               N/A
GPU7    SYS     SYS     SYS     SYS     SYS     SYS     PIX      X      24-47   1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

The command I use :

docker run --rm -it --runtime nvidia \
-e CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
-e NVIDIA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
--gpus all \
-v /data/vllm/huggingface:/root/vllm/huggingface \
-v /data/models/mistral/mistral-large-instruct-2407:/root/data/mistral-large-instruct-2407 \
-p 8090:8000 \
--ipc=host \
--name vllm-mistral \
vllm/vllm-openai:v0.5.4 \
--model /root/data/mistral-large-instruct-2407 \
--served-model-name general \
--load-format safetensors \
--distributed-executor-backend ray \
--quantization fp8 \
--disable-custom-all-reduce \
--tensor-parallel-size 8 \
--trust-remote-code \
--max-num-seqs 32 \
--kv-cache-dtype fp8 \
--enable-chunked-prefill=False

Any guidance or suggestions would be greatly appreciated!