[Bug]: Multi-GPU AMD Setup Hangs without NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce

[taddeusb90]

Your current environment

The output of `python collect_env.py` plus rest of the environment if it might prove useful

+ python collect_env.py
DEBUG 01-27 18:16:52 __init__.py:26] No plugins for group vllm.platform_plugins found.
INFO 01-27 18:16:52 __init__.py:183] Automatically detected platform rocm.
Collecting environment information...
PyTorch version: 2.6.0a0+git8d4926e
Is debug build: False
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: 6.3.42133-1b9c17779

OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: 18.0.0git (https://github.com/RadeonOpenCompute/llvm-project roc-6.3.1 24491 1e0fda770a2079fbd71e4b70974d74f62fd3af10)
CMake version: version 3.31.4
Libc version: glibc-2.35

Python version: 3.12.8 (main, Dec  4 2024, 08:54:12) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-125-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: AMD Instinct MI210 (gfx90a:sramecc+:xnack-)
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 6.3.42133
MIOpen runtime version: 3.3.0
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        43 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               128
On-line CPU(s) list:                  0-127
Vendor ID:                            AuthenticAMD
Model name:                           AMD EPYC 7542 32-Core Processor
CPU family:                           23
Model:                                49
Thread(s) per core:                   2
Core(s) per socket:                   32
Socket(s):                            2
Stepping:                             0
Frequency boost:                      enabled
CPU max MHz:                          2900.0000
CPU min MHz:                          1500.0000
BogoMIPS:                             5799.35
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es
Virtualization:                       AMD-V
L1d cache:                            2 MiB (64 instances)
L1i cache:                            2 MiB (64 instances)
L2 cache:                             32 MiB (64 instances)
L3 cache:                             256 MiB (16 instances)
NUMA node(s):                         2
NUMA node0 CPU(s):                    0-31,64-95
NUMA node1 CPU(s):                    32-63,96-127
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow:   Mitigation; safe RET
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] pyzmq==26.2.0
[pip3] torch==2.6.0a0+git8d4926e
[pip3] torchvision==0.19.1a0+6194369
[pip3] transformers==4.48.1
[pip3] triton==3.2.0+gite5be006a
[conda] Could not collect
ROCM Version: 6.3.42133-1b9c17779
Neuron SDK Version: N/A
vLLM Version: 0.6.4.post2.dev822+g16366ee8b
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
============================ ROCm System Management Interface ============================
================================ Weight between two GPUs =================================
       GPU0         GPU1         
GPU0   0            40           
GPU1   40           0            

================================= Hops between two GPUs ==================================
       GPU0         GPU1         
GPU0   0            2            
GPU1   2            0            

=============================== Link Type between two GPUs ===============================
       GPU0         GPU1         
GPU0   0            PCIE         
GPU1   PCIE         0            

======================================= Numa Nodes =======================================
GPU[0]		: (Topology) Numa Node: 0
GPU[0]		: (Topology) Numa Affinity: 0
GPU[1]		: (Topology) Numa Node: 0
GPU[1]		: (Topology) Numa Affinity: 0
================================== End of ROCm SMI Log ===================================

TORCH_USE_HIP_DSA=1
NCCL_P2P_DISABLE=1
NCCL_DEBUG=TRACE
VLLM_WORKER_MULTIPROC_METHOD=spawn
VLLM_TRACE_FUNCTION=1
PYTORCH_ROCM_ARCH=gfx90a;gfx942
LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/cv2/../../lib64:/opt/rocm/lib:/usr/local/lib:
VLLM_LOGGING_LEVEL=DEBUG
VLLM_USE_TRITON_FLASH_ATTN=0
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

+ echo '=== Launching vLLM in background with real-time logs ==='
+ echo 'Running vLLM with model=Qwen/QwQ-32B-preview'
=== Launching vLLM in background with real-time logs ===
Running vLLM with model=Qwen/QwQ-32B-preview
+ vllm serve Qwen/QwQ-32B-preview --host=0.0.0.0 --port=8000 --max-model-len=2048 --enforce-eager --dtype=half --tensor-parallel-size 2 --disable-custom-all-reduce --trust-remote-code
DEBUG 01-27 18:17:13 __init__.py:26] No plugins for group vllm.platform_plugins found.
INFO 01-27 18:17:13 __init__.py:183] Automatically detected platform rocm.
INFO 01-27 18:17:14 api_server.py:768] vLLM API server version 0.6.4.post2.dev822+g16366ee8b
INFO 01-27 18:17:14 api_server.py:769] args: Namespace(subparser='serve', model_tag='Qwen/QwQ-32B-preview', config='', host='0.0.0.0', port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='Qwen/QwQ-32B-preview', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='half', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=2048, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=True, max_seq_len_to_capture=8192, disable_custom_all_reduce=True, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, enable_sleep_mode=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function serve at 0x7f9ef9a439c0>)
DEBUG 01-27 18:17:14 __init__.py:26] No plugins for group vllm.general_plugins found.
DEBUG 01-27 18:17:14 api_server.py:176] Multiprocessing frontend to use ipc:///tmp/ddef63dd-382b-4290-a14d-1a4d58fb1945 for IPC Path.
INFO 01-27 18:17:14 api_server.py:195] Started engine process with PID 855
WARNING 01-27 18:17:16 config.py:2325] Casting torch.bfloat16 to torch.float16.
DEBUG 01-27 18:17:18 __init__.py:26] No plugins for group vllm.platform_plugins found.
INFO 01-27 18:17:19 __init__.py:183] Automatically detected platform rocm.
DEBUG 01-27 18:17:19 __init__.py:26] No plugins for group vllm.general_plugins found.
WARNING 01-27 18:17:22 config.py:2325] Casting torch.bfloat16 to torch.float16.
INFO 01-27 18:17:38 config.py:528] This model supports multiple tasks: {'score', 'embed', 'classify', 'reward', 'generate'}. Defaulting to 'generate'.
INFO 01-27 18:17:38 config.py:1335] Defaulting to use mp for distributed inference
INFO 01-27 18:17:38 config.py:1365] Disabled the custom all-reduce kernel because it is not supported on AMD GPUs.
WARNING 01-27 18:17:38 rocm.py:109] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
WARNING 01-27 18:17:38 config.py:664] Async output processing is not supported on the current platform type cuda.
INFO 01-27 18:17:42 config.py:528] This model supports multiple tasks: {'reward', 'score', 'embed', 'classify', 'generate'}. Defaulting to 'generate'.
INFO 01-27 18:17:42 config.py:1335] Defaulting to use mp for distributed inference
INFO 01-27 18:17:42 config.py:1365] Disabled the custom all-reduce kernel because it is not supported on AMD GPUs.
WARNING 01-27 18:17:42 rocm.py:109] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
WARNING 01-27 18:17:42 config.py:664] Async output processing is not supported on the current platform type cuda.
INFO 01-27 18:17:42 llm_engine.py:232] Initializing an LLM engine (v0.6.4.post2.dev822+g16366ee8b) with config: model='Qwen/QwQ-32B-preview', speculative_config=None, tokenizer='Qwen/QwQ-32B-preview', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen/QwQ-32B-preview, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"candidate_compile_sizes":[],"compile_sizes":[],"capture_sizes":[],"max_capture_size":0}, use_cached_outputs=True, 
WARNING 01-27 18:17:43 multiproc_worker_utils.py:298] Reducing Torch parallelism from 128 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 01-27 18:17:43 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
WARNING 01-27 18:17:43 logger.py:201] VLLM_TRACE_FUNCTION is enabled. It will record every function executed by Python. This will slow down the code. It is suggested to be used for debugging hang or crashes only.
INFO 01-27 18:17:43 logger.py:205] Trace frame log is saved to /tmp/root/vllm/vllm-instance-c1065/VLLM_TRACE_FUNCTION_for_process_855_thread_139818853159104_at_2025-01-27_18:17:43.312488.log
DEBUG 01-27 18:17:47 __init__.py:26] No plugins for group vllm.platform_plugins found.
INFO 01-27 18:17:47 __init__.py:183] Automatically detected platform rocm.
(VllmWorkerProcess pid=1318) INFO 01-27 18:17:48 multiproc_worker_utils.py:227] Worker ready; awaiting tasks
(VllmWorkerProcess pid=1318) WARNING 01-27 18:17:48 logger.py:201] VLLM_TRACE_FUNCTION is enabled. It will record every function executed by Python. This will slow down the code. It is suggested to be used for debugging hang or crashes only.
(VllmWorkerProcess pid=1318) INFO 01-27 18:17:48 logger.py:205] Trace frame log is saved to /tmp/root/vllm/vllm-instance-c1065/VLLM_TRACE_FUNCTION_for_process_1318_thread_140020751671488_at_2025-01-27_18:17:48.631413.log
(VllmWorkerProcess pid=1318) DEBUG 01-27 18:17:48 __init__.py:26] No plugins for group vllm.general_plugins found.
DEBUG 01-27 18:17:51 client.py:188] Waiting for output from MQLLMEngine.
DEBUG 01-27 18:18:01 client.py:188] Waiting for output from MQLLMEngine.
DEBUG 01-27 18:18:11 client.py:188] Waiting for output from MQLLMEngine.

Added full log for more context at runtime.
full-log.txt

Model Input Dumps

No response

🐛 Describe the bug

Hello ROCm Team,

I’ve been trying to run vLLM on multi-GPU cluster for the purpose of this bug I'll share a case with two AMD MI210 GPUs (ROCm environment). We have a Kubernetes cluster where each node provides two AMD GPUs, and everything works fine on a single GPU. However, as soon as I enable multi‐GPU (using --tensor-parallel-size 2), vLLM hangs and never fully starts the server unless I set:

export NCCL_P2P_DISABLE=1

With NCCL_P2P_DISABLE=1, the multi‐GPU server initializes successfully, but this obviously forces GPU‐to‐GPU traffic to go through host memory (slower than direct peer‐to‐peer).

Symptoms

• Single GPU: Works well with most tested models.
• Two GPUs (tensor parallel = 2): Service hangs at RCCL init stage. No further logs; never sees the “Started server process” message.
• Workaround: NCCL_P2P_DISABLE=1 unblocks the hang and I can run multi‐GPU inference, but throughput is presumably slower.

Test Matrix & Observations

I tested multiple environment variables (HIP_FORCE_P2P_HOST, HSA_ENABLE_SDMA, etc.) plus variations of flash-attention, with or without cuMask, etc. Some combos succeed on many 7B–20B models, but none allowed multi‐GPU to start without disabling P2P.
I also see kernel logs indicating that iommu=pt is not set and kernel.numa_balancing is 1. Currently, we can’t fix that on the node’s kernel. Possibly that’s the underlying cause of the hang. But the net effect is: multi‐GPU gets stuck unless P2P is turned off.

Below is a snippet of my job YAML (Kubernetes) that includes:

export HIP_VISIBLE_DEVICES=0,1
export ROCR_VISIBLE_DEVICES=0,1
export NCCL_P2P_DISABLE=1   # The only way to get multi-GPU working
vllm serve ...
  --tensor-parallel-size 2
  --disable-custom-all-reduce

And it starts up fine, whereas without NCCL_P2P_DISABLE=1 it just hangs forever at RCCL init.

Questions

1. Recommended Settings

• Are there official or recommended environment variables to ensure multi‐GPU AMD ROCm works with vLLM, especially if we cannot modify host kernel parameters like iommu=pt and kernel.numa_balancing=0?
• Is there a “one size fits all” environment variable set that you suggest for AMD GPUs to avoid P2P or specific issues?
• Is there an official or recommended setup? (including kernel changes)

2. Kernel‐Level Requirements

• Do you suggest we absolutely must set iommu=pt and disable NUMA auto‐balancing on the host for stable multi‐GPU? Or are there known overrides in RCCL/vLLM for multi‐GPU ROCm that can achieve stable operation without those kernel changes?

3. Performance vs. Stability

• Is there any guidance on which env vars or vLLM flags best preserve performance if we can’t set iommu=pt? We’d prefer not to rely on NCCL_P2P_DISABLE=1 if there’s a less costly fallback.
• Thanks in advance for any advice. The single‐GPU experience is great; multi‐GPU is the only stumbling block for us right now. If you have any insights or updated best‐practice docs, please let me know!

Logs & Additional Data

• We have run a fairly broad matrix of environment variables:
  HIP_FORCE_P2P_HOST, HSA_ENABLE_SDMA, ROCM_DISABLE_CU_MASK, TORCH_USE_HIP_DSA, VLLM_USE_TRITON_FLASH_ATTN, etc.
• In all cases, multi‐GPU only works if we fully disable GPU P2P.
• Single GPU usage is rock‐solid with all tested models.

Let us know if you have recommended kernel settings or any user‐space environment variables that typically solve this in AMD ROCm + vLLM.
Thank you!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ppanchad-amd]

Hi @taddeusb90. Internal ticket has been created to investigate this issue. Thanks!

[schung-amd]

Hi @taddeusb90, sorry for the delay on this. I'll try to address your questions in a broad sense, and we can narrow in on the details.

Are there official or recommended environment variables to ensure multi‐GPU AMD ROCm works with vLLM, especially if we cannot modify host kernel parameters like iommu=pt and kernel.numa_balancing=0?

Is there a “one size fits all” environment variable set that you suggest for AMD GPUs to avoid P2P or specific issues?

The RCCL docs suggest that HSA_FORCE_FINE_GRAIN_PCIE=1 is required for P2P with PCIE-connected GPUs, along with large BAR addressing support. I'm not aware of any environment variables that are required to be set for vLLM specifically. RCCL does support various other P2P-related environment variables, but it's unclear which would help in this case and we don't have good external documentation; I'll look into these and get back to you.

Is there an official or recommended setup? (including kernel changes)

We have some validated configurations in the docs. For example, the MI200 settings are at https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi200.html. I'd also suggest checking if ACS is disabled: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs.

Do you suggest we absolutely must set iommu=pt and disable NUMA auto‐balancing on the host for stable multi‐GPU? Or are there known overrides in RCCL/vLLM for multi‐GPU ROCm that can achieve stable operation without those kernel changes?

Is there any guidance on which env vars or vLLM flags best preserve performance if we can’t set iommu=pt? We’d prefer not to rely on NCCL_P2P_DISABLE=1 if there’s a less costly fallback.

iommu=pt might be required if IOMMU is enabled in BIOS, disabling NUMA balancing shouldn't be necessary but might improve performance. The issue here seems to be P2P not working, so any workaround would likely involve disabling P2P.

In addition to the above, did you build torch from source? If not, you should use torch wheels for ROCm, either wheels from pytorch.org suffixed with +ROCm or wheels from repo.radeon.com/rocm/manylinux. I would also suggest making sure vLLM is up to date. If you could also provide the program output leading up to the hang with NCCL_DEBUG=TRACE that would be helpful.

I'm still gathering info on this. We have looked into similar issues in recent months, but none with a clean answer that looks applicable to this. I'll update when I have more information.

[schung-amd]

@taddeusb90 Are you still experiencing this issue? If so, can you provide the program output leading up to the hang with NCCL_DEBUG=TRACE set?

[DrZoidberg09]

@schung-amd I am experiencing exactly the same issue. It works with disabled P2P, but crashes when enabled. IOMMU is disabled, ReBAR is enabled, ROCm 6.4 with dual gfx1100 with EPYC Siena. (can deliver further information about the setup if desired).

Here is the log:

root@supllm:/app#vllm serve ./model/Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf --tensor-parallel-size 2 --max-model-len 32000 --gpu-memory-utilization 0.8 --chat-template ./model/chat_template.jinja INFO 04-28 12:30:03 [__init__.py:239] Automatically detected platform rocm. INFO 04-28 12:30:16 [api_server.py:1043] vLLM API server version 0.8.5.dev178+g3cde34a4a INFO 04-28 12:30:16 [api_server.py:1044] args: Namespace(subparser='serve', model_tag='./model/Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf', config='', host=None, port=8000, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template='./model/chat_template.jinja', chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='./model/Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', max_model_len=32000, guided_decoding_backend='auto', reasoning_parser=None, logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=2, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_memory_utilization=0.8, swap_space=4, kv_cache_dtype='auto', num_gpu_blocks_override=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, disable_sliding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, speculative_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, enable_chunked_prefill=None, multi_step_stream_outputs=True, scheduling_policy='fcfs', disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=False, disable_cascade_attn=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False, dispatch_function=<function ServeSubcommand.cmd at 0x7db6e123bc40>) INFO 04-28 12:30:36 [config.py:2941] Downcasting torch.float32 to torch.float16. INFO 04-28 12:30:53 [config.py:718] This model supports multiple tasks: {'reward', 'generate', 'classify', 'score', 'embed'}. Defaulting to 'generate'. WARNING 04-28 12:30:53 [config.py:798] gguf quantization is not fully optimized yet. The speed can be slower than non-quantized models. WARNING 04-28 12:30:53 [arg_utils.py:1695] --quantization gguf is not supported by the V1 Engine. Falling back to V0. INFO 04-28 12:30:53 [config.py:1784] Defaulting to use mp for distributed inference INFO 04-28 12:30:53 [config.py:1818] Disabled the custom all-reduce kernel because it is not supported on current platform. INFO 04-28 12:30:53 [api_server.py:246] Started engine process with PID 402 INFO 04-28 12:30:57 [__init__.py:239] Automatically detected platform rocm. You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that thelegacy(previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, setlegacy=False. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message. INFO 04-28 12:31:09 [llm_engine.py:243] Initializing a V0 LLM engine (v0.8.5.dev178+g3cde34a4a) with config: model='./model/Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf', speculative_config=None, tokenizer='./model/Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32000, download_dir=None, load_format=LoadFormat.GGUF, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=gguf, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=./model/Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True, You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the legacy(previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, setlegacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in huggingface/transformers#24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message.
WARNING 04-28 12:31:49 [multiproc_worker_utils.py:306] Reducing Torch parallelism from 48 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 04-28 12:31:49 [rocm.py:186] None is not supported in AMD GPUs.
INFO 04-28 12:31:49 [rocm.py:187] Using ROCmFlashAttention backend.
INFO 04-28 12:31:52 [init.py:239] Automatically detected platform rocm.
(VllmWorkerProcess pid=461) INFO 04-28 12:32:04 [multiproc_worker_utils.py:225] Worker ready; awaiting tasks
(VllmWorkerProcess pid=461) INFO 04-28 12:32:04 [rocm.py:186] None is not supported in AMD GPUs.
(VllmWorkerProcess pid=461) INFO 04-28 12:32:04 [rocm.py:187] Using ROCmFlashAttention backend.
(VllmWorkerProcess pid=461) INFO 04-28 12:32:05 [utils.py:1043] Found nccl from library librccl.so.1
(VllmWorkerProcess pid=461) INFO 04-28 12:32:05 [pynccl.py:69] vLLM is using nccl==2.22.3
INFO 04-28 12:32:05 [utils.py:1043] Found nccl from library librccl.so.1
INFO 04-28 12:32:05 [pynccl.py:69] vLLM is using nccl==2.22.3
supllm:402:402 [0] NCCL INFO Kernel version: 6.8.0-58-generic

supllm:402:402 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/init.cc:179 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
supllm:402:402 [0] NCCL INFO Disabled GDRCopy equivalent memory allocation on gfx1100 due to GPU architecture
supllm:402:402 [0] NCCL INFO Bootstrap : Using enp203s0:192.168.100.86<0>
supllm:461:461 [1] NCCL INFO ROCr version 1.15
supllm:461:461 [1] NCCL INFO Dmabuf feature disabled without NCCL_DMABUF_ENABLE=1
supllm:402:402 [0] NCCL INFO ROCr version 1.15
supllm:461:461 [1] NCCL INFO Kernel version: 6.8.0-58-generic
supllm:402:402 [0] NCCL INFO Dmabuf feature disabled without NCCL_DMABUF_ENABLE=1

supllm:461:461 [1] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/init.cc:179 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
supllm:461:461 [1] NCCL INFO Disabled GDRCopy equivalent memory allocation on gfx1100 due to GPU architecture
supllm:402:402 [0] NCCL INFO Kernel version: 6.8.0-58-generic

supllm:402:402 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/init.cc:179 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
supllm:402:402 [0] NCCL INFO RCCL version : 2.22.3-HEAD:271856d
HIP version : 6.4.43482-0f2d60242
ROCm version : 6.4.0.0-47-73dae9c
Hostname : supllm
Librccl path : /opt/rocm/lib/librccl.so.1
supllm:461:461 [1] NCCL INFO Bootstrap : Using enp203s0:192.168.100.86<0>
supllm:461:461 [1] NCCL INFO RCCL version : 2.22.3-HEAD:271856d
HIP version : 6.4.43482-0f2d60242
ROCm version : 6.4.0.0-47-73dae9c
Hostname : supllm
Librccl path : /opt/rocm/lib/librccl.so.1
supllm:461:461 [1] NCCL INFO NET/Plugin: Could not find: librccl-net.so. Using internal network plugin.
supllm:461:461 [1] NCCL INFO Failed to open libibverbs.so[.1]
supllm:461:461 [1] NCCL INFO NET/Socket : Using [0]enp203s0:192.168.100.86<0> [1]br-03e037269530:172.19.0.1<0> [2]veth18cd3f2:fe80::1453:a2ff:fede:4989%veth18cd3f2<0> [3]vetha4dfd7e:fe80::8800:b0ff:fee0:6283%vetha4dfd7e<0> [4]veth442b76f:fe80::8c9c:e3ff:fe1c:4252%veth442b76f<0> [5]vethcaa0edd:fe80::7a:44ff:fe32:21ab%vethcaa0edd<0> [6]veth983f9f6:fe80::8051:25ff:fe0a:990c%veth983f9f6<0>
supllm:461:461 [1] NCCL INFO Using network Socket
supllm:461:461 [1] NCCL INFO [node_id = 2; gpu_id = 45048; unique_id = 5402358709874111910; location_id = 17152; bdf = 17152; domain = 0; partition = 0],
supllm:461:461 [1] NCCL INFO [node_id = 1; gpu_id = 57594; unique_id = 5756466707879984034; location_id = 49920; bdf = 49920; domain = 0; partition = 0],
supllm:461:461 [1] NCCL INFO initialized internal alternative rsmi functionality
supllm:402:402 [0] NCCL INFO NET/Plugin: Could not find: librccl-net.so. Using internal network plugin.
supllm:402:402 [0] NCCL INFO Failed to open libibverbs.so[.1]
supllm:402:402 [0] NCCL INFO NET/Socket : Using [0]enp203s0:192.168.100.86<0> [1]br-03e037269530:172.19.0.1<0> [2]veth18cd3f2:fe80::1453:a2ff:fede:4989%veth18cd3f2<0> [3]vetha4dfd7e:fe80::8800:b0ff:fee0:6283%vetha4dfd7e<0> [4]veth442b76f:fe80::8c9c:e3ff:fe1c:4252%veth442b76f<0> [5]vethcaa0edd:fe80::7a:44ff:fe32:21ab%vethcaa0edd<0> [6]veth983f9f6:fe80::8051:25ff:fe0a:990c%veth983f9f6<0>
supllm:402:402 [0] NCCL INFO Using network Socket
supllm:402:402 [0] NCCL INFO [node_id = 2; gpu_id = 45048; unique_id = 5402358709874111910; location_id = 17152; bdf = 17152; domain = 0; partition = 0],
supllm:402:402 [0] NCCL INFO [node_id = 1; gpu_id = 57594; unique_id = 5756466707879984034; location_id = 49920; bdf = 49920; domain = 0; partition = 0],
supllm:402:402 [0] NCCL INFO initialized internal alternative rsmi functionality
supllm:402:402 [0] NCCL INFO ncclCommInitRank comm 0x4ecb0890 rank 0 nranks 2 cudaDev 0 nvmlDev 1 busId c3000 commId 0x2a3913dce99a68df - Init START
supllm:461:461 [1] NCCL INFO ncclCommInitRank comm 0x540e2310 rank 1 nranks 2 cudaDev 1 nvmlDev 0 busId 43000 commId 0x2a3913dce99a68df - Init START
supllm:402:402 [0] NCCL INFO initialized internal alternative rsmi functionality
supllm:461:461 [1] NCCL INFO initialized internal alternative rsmi functionality
supllm:402:402 [0] NCCL INFO Setting affinity for GPU 1 to ffff,ffffffff
supllm:461:461 [1] NCCL INFO Setting affinity for GPU 0 to ffff,ffffffff
supllm:461:461 [1] NCCL INFO comm 0x540e2310 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
supllm:402:402 [0] NCCL INFO comm 0x4ecb0890 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
supllm:461:461 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 comm 0x540e2310 nRanks 02 busId 43000
supllm:402:402 [0] NCCL INFO Channel 00/02 : 0 1
supllm:461:461 [1] NCCL INFO P2P Chunksize set to 131072
supllm:402:402 [0] NCCL INFO Channel 01/02 : 0 1
supllm:402:402 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 comm 0x4ecb0890 nRanks 02 busId c3000
supllm:402:402 [0] NCCL INFO P2P Chunksize set to 131072
supllm:402:503 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/transport/p2p.cc:239 NCCL WARN hipIpcGetMemHandle failed : invalid argument
supllm:402:503 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/transport/p2p.cc:241 NCCL WARN Cuda failure 'invalid argument'
supllm:402:503 [0] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/transport/p2p.cc:652 -> 1
supllm:402:402 [0] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/transport/p2p.cc:477 -> 1
supllm:402:402 [0] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/transport.cc:46 -> 1
supllm:402:402 [0] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/transport.cc:148 -> 1
supllm:402:402 [0] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/transport/generic.cc:11 -> 1
supllm:402:402 [0] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/init.cc:1633 -> 1
supllm:402:402 [0] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/init.cc:1906 -> 1
supllm:402:402 [0] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/init.cc:2218 -> 1
supllm:461:502 [1] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/transport/p2p.cc:239 NCCL WARN hipIpcGetMemHandle failed : invalid argument
supllm:402:402 [0] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/init.cc:2257 -> 1
supllm:461:502 [1] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/transport/p2p.cc:241 NCCL WARN Cuda failure 'invalid argument'
supllm:461:502 [1] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/transport/p2p.cc:652 -> 1
supllm:461:461 [1] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/transport/p2p.cc:477 -> 1
supllm:461:461 [1] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/transport.cc:46 -> 1
supllm:461:461 [1] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/transport.cc:148 -> 1
supllm:461:461 [1] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/transport/generic.cc:11 -> 1
supllm:461:461 [1] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/init.cc:1633 -> 1
supllm:461:461 [1] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/init.cc:1906 -> 1
supllm:461:461 [1] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/init.cc:2218 -> 1
supllm:461:461 [1] NCCL INFO /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/init.cc:2257 -> 1
ERROR 04-28 12:32:06 [engine.py:448] NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
ERROR 04-28 12:32:06 [engine.py:448] Traceback (most recent call last):
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 436, in run_mp_engine
ERROR 04-28 12:32:06 [engine.py:448] engine = MQLLMEngine.from_vllm_config(
ERROR 04-28 12:32:06 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 128, in from_vllm_config
ERROR 04-28 12:32:06 [engine.py:448] return cls(
ERROR 04-28 12:32:06 [engine.py:448] ^^^^
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 82, in init
ERROR 04-28 12:32:06 [engine.py:448] self.engine = LLMEngine(*args, **kwargs)
ERROR 04-28 12:32:06 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 282, in init
ERROR 04-28 12:32:06 [engine.py:448] self.model_executor = executor_class(vllm_config=vllm_config, )
ERROR 04-28 12:32:06 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 286, in init
ERROR 04-28 12:32:06 [engine.py:448] super().init(*args, **kwargs)
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 52, in init
ERROR 04-28 12:32:06 [engine.py:448] self._init_executor()
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/mp_distributed_executor.py", line 124, in _init_executor
ERROR 04-28 12:32:06 [engine.py:448] self._run_workers("init_device")
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/mp_distributed_executor.py", line 185, in _run_workers
ERROR 04-28 12:32:06 [engine.py:448] driver_worker_output = run_method(self.driver_worker, sent_method,
ERROR 04-28 12:32:06 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2428, in run_method
ERROR 04-28 12:32:06 [engine.py:448] return func(*args, **kwargs)
ERROR 04-28 12:32:06 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 604, in init_device
ERROR 04-28 12:32:06 [engine.py:448] self.worker.init_device() # type: ignore
ERROR 04-28 12:32:06 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 186, in init_device
ERROR 04-28 12:32:06 [engine.py:448] init_worker_distributed_environment(self.vllm_config, self.rank,
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 527, in init_worker_distributed_environment
ERROR 04-28 12:32:06 [engine.py:448] ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 964, in ensure_model_parallel_initialized
ERROR 04-28 12:32:06 [engine.py:448] initialize_model_parallel(tensor_model_parallel_size,
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 917, in initialize_model_parallel
ERROR 04-28 12:32:06 [engine.py:448] _TP = init_model_parallel_group(group_ranks,
ERROR 04-28 12:32:06 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 730, in init_model_parallel_group
ERROR 04-28 12:32:06 [engine.py:448] return GroupCoordinator(
ERROR 04-28 12:32:06 [engine.py:448] ^^^^^^^^^^^^^^^^^
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 206, in init
ERROR 04-28 12:32:06 [engine.py:448] self.device_communicator = device_comm_cls(
ERROR 04-28 12:32:06 [engine.py:448] ^^^^^^^^^^^^^^^^
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 39, in init
ERROR 04-28 12:32:06 [engine.py:448] self.pynccl_comm = PyNcclCommunicator(
ERROR 04-28 12:32:06 [engine.py:448] ^^^^^^^^^^^^^^^^^^^
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 99, in init
ERROR 04-28 12:32:06 [engine.py:448] self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
ERROR 04-28 12:32:06 [engine.py:448] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 277, in ncclCommInitRank
ERROR 04-28 12:32:06 [engine.py:448] self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),
ERROR 04-28 12:32:06 [engine.py:448] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 256, in NCCL_CHECK
ERROR 04-28 12:32:06 [engine.py:448] raise RuntimeError(f"NCCL error: {error_str}")
ERROR 04-28 12:32:06 [engine.py:448] RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] Exception in worker VllmWorkerProcess while processing method init_device.
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] Traceback (most recent call last):
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/multiproc_worker_utils.py", line 232, in _run_worker_process
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] output = run_method(worker, method, args, kwargs)
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2428, in run_method
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] return func(*args, **kwargs)
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 604, in init_device
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] self.worker.init_device() # type: ignore
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 186, in init_device
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] init_worker_distributed_environment(self.vllm_config, self.rank,
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 527, in init_worker_distributed_environment
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 964, in ensure_model_parallel_initialized
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] initialize_model_parallel(tensor_model_parallel_size,
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 917, in initialize_model_parallel
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] _TP = init_model_parallel_group(group_ranks,
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 730, in init_model_parallel_group
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] return GroupCoordinator(
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 206, in init
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] self.device_communicator = device_comm_cls(
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 39, in init
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] self.pynccl_comm = PyNcclCommunicator(
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 99, in init
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 277, in ncclCommInitRank
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 256, in NCCL_CHECK
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] raise RuntimeError(f"NCCL error: {error_str}")
(VllmWorkerProcess pid=461) ERROR 04-28 12:32:06 [multiproc_worker_utils.py:238] RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
Process SpawnProcess-1:
ERROR 04-28 12:32:06 [multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 461 died, exit code: -15
INFO 04-28 12:32:06 [multiproc_worker_utils.py:124] Killing local vLLM worker processes
Traceback (most recent call last):
File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 450, in run_mp_engine
raise e
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 436, in run_mp_engine
engine = MQLLMEngine.from_vllm_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 128, in from_vllm_config
return cls(
^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 82, in init
self.engine = LLMEngine(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 282, in init
self.model_executor = executor_class(vllm_config=vllm_config, )
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 286, in init
super().init(*args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 52, in init
self._init_executor()
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/mp_distributed_executor.py", line 124, in _init_executor
self._run_workers("init_device")
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/mp_distributed_executor.py", line 185, in _run_workers
driver_worker_output = run_method(self.driver_worker, sent_method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2428, in run_method
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 604, in init_device
self.worker.init_device() # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 186, in init_device
init_worker_distributed_environment(self.vllm_config, self.rank,
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 527, in init_worker_distributed_environment
ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 964, in ensure_model_parallel_initialized
initialize_model_parallel(tensor_model_parallel_size,
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 917, in initialize_model_parallel
_TP = init_model_parallel_group(group_ranks,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 730, in init_model_parallel_group
return GroupCoordinator(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 206, in init
self.device_communicator = device_comm_cls(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 39, in init
self.pynccl_comm = PyNcclCommunicator(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 99, in init
self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 277, in ncclCommInitRank
self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 256, in NCCL_CHECK
raise RuntimeError(f"NCCL error: {error_str}")
RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
[rank0]:[W428 12:32:06.780430748 ProcessGroupNCCL.cpp:1476] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
Traceback (most recent call last):
File "/usr/local/bin/vllm", line 8, in
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 53, in main
args.dispatch_function(args)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 27, in cmd
uvloop.run(run_server(args))
File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 109, in run
return __asyncio.run(
^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 61, in wrapper
return await main
^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1078, in run_server
async with build_async_engine_client(args) as engine_client:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 269, in build_async_engine_client_from_engine_args
raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.

[schung-amd]

Thanks @DrZoidberg09, I'll take a look.

[wenhui-ml]

@schung-amd Hi，has this issue been solved?

[wenhui-ml]

I use the latest docker image fo rocm/vllm pulled form docker hub, it still run with this error.
"root@chenwh:/app/model# vllm serve ./gemma-3-4b-it --tensor-parallel-size 2 --max-model-len 65536 --port 8120 --trust-remote-code --served-model-name gemma3-4b --enable-chunked-prefill --max-num-batched-tokens 2048 --gpu-memory-utilization 0.90
INFO 06-18 16:34:56 [init.py:248] Automatically detected platform rocm.
INFO 06-18 16:35:00 [init.py:30] Available plugins for group vllm.general_plugins:
INFO 06-18 16:35:00 [init.py:32] name=lora_filesystem_resolver, value=vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 06-18 16:35:00 [init.py:34] all available plugins for group vllm.general_plugins will be loaded.
INFO 06-18 16:35:00 [init.py:36] set environment variable VLLM_PLUGINS to control which plugins to load.
INFO 06-18 16:35:00 [init.py:44] plugin lora_filesystem_resolver loaded.
INFO 06-18 16:35:01 [config.py:1915] Disabled the custom all-reduce kernel because it is not supported on current platform.
INFO 06-18 16:35:01 [config.py:1886] Disabled the custom all-reduce kernel because it is not working correctly when using two AMD Navi GPUs.
INFO 06-18 16:35:01 [config.py:1915] Disabled the custom all-reduce kernel because it is not supported on current platform.
INFO 06-18 16:35:01 [config.py:1886] Disabled the custom all-reduce kernel because it is not working correctly when using two AMD Navi GPUs.
INFO 06-18 16:35:01 [config.py:1915] Disabled the custom all-reduce kernel because it is not supported on current platform.
INFO 06-18 16:35:01 [config.py:1886] Disabled the custom all-reduce kernel because it is not working correctly when using two AMD Navi GPUs.
INFO 06-18 16:35:01 [api_server.py:1289] vLLM API server version 0.8.6.dev315+g91a560098
INFO 06-18 16:35:01 [config.py:1915] Disabled the custom all-reduce kernel because it is not supported on current platform.
INFO 06-18 16:35:01 [config.py:1886] Disabled the custom all-reduce kernel because it is not working correctly when using two AMD Navi GPUs.
INFO 06-18 16:35:01 [cli_args.py:300] non-default args: {'port': 8120, 'trust_remote_code': True, 'max_model_len': 65536, 'served_model_name': ['gemma3-4b'], 'tensor_parallel_size': 2, 'max_num_batched_tokens': 2048, 'enable_chunked_prefill': True}
INFO 06-18 16:35:08 [config.py:788] This model supports multiple tasks: {'score', 'reward', 'generate', 'classify', 'embed'}. Defaulting to 'generate'.
INFO 06-18 16:35:08 [arg_utils.py:1601] rocm is experimental on VLLM_USE_V1=1. Falling back to V0 Engine.
INFO 06-18 16:35:08 [config.py:1870] Defaulting to use mp for distributed inference
INFO 06-18 16:35:08 [config.py:1915] Disabled the custom all-reduce kernel because it is not supported on current platform.
INFO 06-18 16:35:08 [config.py:1886] Disabled the custom all-reduce kernel because it is not working correctly when using two AMD Navi GPUs.
INFO 06-18 16:35:08 [config.py:2124] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 06-18 16:35:08 [api_server.py:257] Started engine process with PID 974
INFO 06-18 16:35:10 [init.py:248] Automatically detected platform rocm.
INFO 06-18 16:35:14 [init.py:30] Available plugins for group vllm.general_plugins:
INFO 06-18 16:35:14 [init.py:32] name=lora_filesystem_resolver, value=vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 06-18 16:35:14 [init.py:34] all available plugins for group vllm.general_plugins will be loaded.
INFO 06-18 16:35:14 [init.py:36] set environment variable VLLM_PLUGINS to control which plugins to load.
INFO 06-18 16:35:14 [init.py:44] plugin lora_filesystem_resolver loaded.
INFO 06-18 16:35:14 [llm_engine.py:240] Initializing a V0 LLM engine (v0.8.6.dev315+g91a560098) with config: model='./gemma-3-4b-it', speculative_config=None, tokenizer='./gemma-3-4b-it', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=65536, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=gemma3-4b, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"compile_sizes": [], "inductor_compile_config": {"enable_auto_functionalized_v2": false}, "cudagraph_capture_sizes": [256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], "max_capture_size": 256}, use_cached_outputs=True,
WARNING 06-18 16:35:15 [multiproc_worker_utils.py:306] Reducing Torch parallelism from 20 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 06-18 16:35:15 [rocm.py:192] None is not supported in AMD GPUs.
INFO 06-18 16:35:15 [rocm.py:193] Using ROCmFlashAttention backend.
INFO 06-18 16:35:17 [init.py:248] Automatically detected platform rocm.
(VllmWorkerProcess pid=1013) INFO 06-18 16:35:21 [multiproc_worker_utils.py:225] Worker ready; awaiting tasks
(VllmWorkerProcess pid=1013) INFO 06-18 16:35:21 [init.py:30] Available plugins for group vllm.general_plugins:
(VllmWorkerProcess pid=1013) INFO 06-18 16:35:21 [init.py:32] name=lora_filesystem_resolver, value=vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
(VllmWorkerProcess pid=1013) INFO 06-18 16:35:21 [init.py:34] all available plugins for group vllm.general_plugins will be loaded.
(VllmWorkerProcess pid=1013) INFO 06-18 16:35:21 [init.py:36] set environment variable VLLM_PLUGINS to control which plugins to load.
(VllmWorkerProcess pid=1013) INFO 06-18 16:35:21 [init.py:44] plugin lora_filesystem_resolver loaded.
(VllmWorkerProcess pid=1013) INFO 06-18 16:35:21 [rocm.py:192] None is not supported in AMD GPUs.
(VllmWorkerProcess pid=1013) INFO 06-18 16:35:21 [rocm.py:193] Using ROCmFlashAttention backend.
(VllmWorkerProcess pid=1013) INFO 06-18 16:35:22 [utils.py:1220] Found nccl from library librccl.so.1
INFO 06-18 16:35:22 [utils.py:1220] Found nccl from library librccl.so.1
(VllmWorkerProcess pid=1013) INFO 06-18 16:35:22 [pynccl.py:69] vLLM is using nccl==2.21.5
INFO 06-18 16:35:22 [pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] Exception in worker VllmWorkerProcess while processing method init_device.
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] Traceback (most recent call last):
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/multiproc_worker_utils.py", line 232, in _run_worker_process
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] output = run_method(worker, method, args, kwargs)
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2769, in run_method
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] return func(*args, **kwargs)
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 604, in init_device
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] self.worker.init_device() # type: ignore
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 218, in init_device
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] init_worker_distributed_environment(self.vllm_config, self.rank,
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 560, in init_worker_distributed_environment
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1102, in ensure_model_parallel_initialized
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] initialize_model_parallel(tensor_model_parallel_size,
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1040, in initialize_model_parallel
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] _TP = init_model_parallel_group(group_ranks,
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 805, in init_model_parallel_group
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] return GroupCoordinator(
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 253, in init
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] self.device_communicator = device_comm_cls(
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 46, in init
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] self.pynccl_comm = PyNcclCommunicator(
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 105, in init
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] self.all_reduce(data)
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 126, in all_reduce
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] self.nccl.ncclAllReduce(buffer_type(in_tensor.data_ptr()),
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 290, in ncclAllReduce
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] self.NCCL_CHECK(self._funcs["ncclAllReduce"](sendbuff, recvbuff, count,
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 256, in NCCL_CHECK
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] raise RuntimeError(f"NCCL error: {error_str}")
(VllmWorkerProcess pid=1013) ERROR 06-18 16:35:22 [multiproc_worker_utils.py:238] RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)

"

[schung-amd]

Sorry, this has not been solved yet. With regards to the original issue, the custom all-reduce barriers in vLLM hang on multi-Radeon systems, so the custom all-reduce is disabled by default. I believe that this deadlock does not occur on NVIDIA due to independent thread scheduling, and does not occur on MI300 due to increased atomic latency. I've been working on an alternative barrier implementation, but thus far my approach is slower than just having custom all-reduce disabled.

The crash you're seeing is unrelated to this though. Can you provide rocminfo output? I haven't been able to reproduce those crashes on dual MI210 or dual gfx1201.

[wenhui-ml]

I use two cards of 7900xtx, gfx1100.

[schung-amd]

Please provide rocminfo output if possible. I don't have access to a dual gfx1100 system at the moment, but I would be surprised if this issue was gfx1100-specific. I've had some systems which did not play nice with the vLLM dockers before, but those typically presented an Invalid device pointer error; still, there may be some hint in the rocminfo output as to similarities between your system and those.

One difference between our systems is that you have a plugin installed; if you disable lora_filesystem_resolver, do you still get this crash?

For reference, the steps I'm taking to try to reproduce this are:

1. Start docker with docker run -it --rm --ipc=host --network=host --group-add video --privileged --security-opt seccomp=unconfined --cap-add=CAP_SYS_ADMIN --cap-add=SYS_PTRACE --device=/dev/kfd --device=/dev/dri rocm/vllm:latest bash
2. Download model with pip install huggingface-hub, huggingface-cli login, huggingface-cli download google/gemma-3-4b-it --local-dir ./model
3. Serve model with vllm serve ./model --tensor-parallel-size 2 --max-model-len 65536 --port 8120 --trust-remote-code --served-model-name gemma3-4b --enable-chunked-prefill --max-num-batched-tokens 2048 --gpu-memory-utilization 0.90

[wenhui-ml]

Hi，this is my rocminfo in docker run with script "docker run -it --rm --ipc=host --network=host --group-add video --privileged --security-opt seccomp=unconfined --cap-add=CAP_SYS_ADMIN --cap-add=SYS_PTRACE --device=/dev/kfd --device=/dev/dri rocm/vllm:latest bash"
"root@chenwh:/app/model# rocminfo
ROCk module version 6.12.12 is loaded

HSA System Attributes

Runtime Version: 1.14
Runtime Ext Version: 1.6
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
Mwaitx: DISABLED
DMAbuf Support: YES

==========
HSA Agents

---

Agent 1

---

Name: Intel(R) Core(TM) i7-14700KF
Uuid: CPU-XX
Marketing Name: Intel(R) Core(TM) i7-14700KF
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 49152(0xc000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 5700
BDFID: 0
Internal Node ID: 0
Compute Unit: 28
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Memory Properties:
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 98711372(0x5e2374c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 98711372(0x5e2374c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 98711372(0x5e2374c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 4
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 98711372(0x5e2374c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

---

Agent 2

---

Name: gfx1100
Uuid: GPU-9af9992c86b1d072
Marketing Name: Radeon RX 7900 XTX
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 1
Device Type: GPU
Cache Info:
L1: 32(0x20) KB
L2: 6144(0x1800) KB
L3: 98304(0x18000) KB
Chip ID: 29772(0x744c)
ASIC Revision: 0(0x0)
Cacheline Size: 128(0x80)
Max Clock Freq. (MHz): 2526
BDFID: 768
Internal Node ID: 1
Compute Unit: 96
SIMDs per CU: 2
Shader Engines: 6
Shader Arrs. per Eng.: 2
WatchPts on Addr. Ranges:4
Coherent Host Access: FALSE
Memory Properties:
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 32(0x20)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 542
SDMA engine uCode:: 24
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 25149440(0x17fc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 25149440(0x17fc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Recommended Granule:0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx1100
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32

---

Agent 3

---

Name: gfx1100
Uuid: GPU-cad49391e695e47e
Marketing Name: Radeon RX 7900 XTX
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 2
Device Type: GPU
Cache Info:
L1: 32(0x20) KB
L2: 6144(0x1800) KB
L3: 98304(0x18000) KB
Chip ID: 29772(0x744c)
ASIC Revision: 0(0x0)
Cacheline Size: 128(0x80)
Max Clock Freq. (MHz): 2526
BDFID: 2560
Internal Node ID: 2
Compute Unit: 96
SIMDs per CU: 2
Shader Engines: 6
Shader Arrs. per Eng.: 2
WatchPts on Addr. Ranges:4
Coherent Host Access: FALSE
Memory Properties:
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 32(0x20)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 542
SDMA engine uCode:: 24
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 25149440(0x17fc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 25149440(0x17fc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Recommended Granule:0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx1100
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32
*** Done ***
root@chenwh:/app/model# "

run with "vllm serve ./model --tensor-parallel-size 2 --max-model-len 65536 --port 8120 --trust-remote-code --served-model-name gemma3-4b --enable-chunked-prefill --max-num-batched-tokens 2048 --gpu-memory-utilization 0.90" is still run with error as below
"root@chenwh:/app/model# vllm serve ./gemma-3-4b-it --tensor-parallel-size 2 --max-model-len 65536 --port 8120 --trust-remote-code --served-model-name gemma3-4b --enable-chunked-prefill --max-num-batched-tokens 2048 --gpu-memory-utilization 0.90
INFO 06-19 14:27:55 [init.py:248] Automatically detected platform rocm.
INFO 06-19 14:28:00 [init.py:30] Available plugins for group vllm.general_plugins:
INFO 06-19 14:28:00 [init.py:32] name=lora_filesystem_resolver, value=vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 06-19 14:28:00 [init.py:34] all available plugins for group vllm.general_plugins will be loaded.
INFO 06-19 14:28:00 [init.py:36] set environment variable VLLM_PLUGINS to control which plugins to load.
INFO 06-19 14:28:00 [init.py:44] plugin lora_filesystem_resolver loaded.
INFO 06-19 14:28:00 [config.py:1915] Disabled the custom all-reduce kernel because it is not supported on current platform.
INFO 06-19 14:28:00 [config.py:1886] Disabled the custom all-reduce kernel because it is not working correctly when using two AMD Navi GPUs.
INFO 06-19 14:28:00 [config.py:1915] Disabled the custom all-reduce kernel because it is not supported on current platform.
INFO 06-19 14:28:00 [config.py:1886] Disabled the custom all-reduce kernel because it is not working correctly when using two AMD Navi GPUs.
INFO 06-19 14:28:00 [config.py:1915] Disabled the custom all-reduce kernel because it is not supported on current platform.
INFO 06-19 14:28:00 [config.py:1886] Disabled the custom all-reduce kernel because it is not working correctly when using two AMD Navi GPUs.
INFO 06-19 14:28:00 [api_server.py:1289] vLLM API server version 0.8.6.dev315+g91a560098
INFO 06-19 14:28:00 [config.py:1915] Disabled the custom all-reduce kernel because it is not supported on current platform.
INFO 06-19 14:28:00 [config.py:1886] Disabled the custom all-reduce kernel because it is not working correctly when using two AMD Navi GPUs.
INFO 06-19 14:28:00 [cli_args.py:300] non-default args: {'port': 8120, 'trust_remote_code': True, 'max_model_len': 65536, 'served_model_name': ['gemma3-4b'], 'tensor_parallel_size': 2, 'max_num_batched_tokens': 2048, 'enable_chunked_prefill': True}
INFO 06-19 14:28:07 [config.py:788] This model supports multiple tasks: {'embed', 'generate', 'reward', 'score', 'classify'}. Defaulting to 'generate'.
INFO 06-19 14:28:07 [arg_utils.py:1601] rocm is experimental on VLLM_USE_V1=1. Falling back to V0 Engine.
INFO 06-19 14:28:07 [config.py:1870] Defaulting to use mp for distributed inference
INFO 06-19 14:28:07 [config.py:1915] Disabled the custom all-reduce kernel because it is not supported on current platform.
INFO 06-19 14:28:07 [config.py:1886] Disabled the custom all-reduce kernel because it is not working correctly when using two AMD Navi GPUs.
INFO 06-19 14:28:07 [config.py:2124] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 06-19 14:28:07 [api_server.py:257] Started engine process with PID 264
INFO 06-19 14:28:09 [init.py:248] Automatically detected platform rocm.
INFO 06-19 14:28:13 [init.py:30] Available plugins for group vllm.general_plugins:
INFO 06-19 14:28:13 [init.py:32] name=lora_filesystem_resolver, value=vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
INFO 06-19 14:28:13 [init.py:34] all available plugins for group vllm.general_plugins will be loaded.
INFO 06-19 14:28:13 [init.py:36] set environment variable VLLM_PLUGINS to control which plugins to load.
INFO 06-19 14:28:13 [init.py:44] plugin lora_filesystem_resolver loaded.
INFO 06-19 14:28:13 [llm_engine.py:240] Initializing a V0 LLM engine (v0.8.6.dev315+g91a560098) with config: model='./gemma-3-4b-it', speculative_config=None, tokenizer='./gemma-3-4b-it', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=65536, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=gemma3-4b, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"compile_sizes": [], "inductor_compile_config": {"enable_auto_functionalized_v2": false}, "cudagraph_capture_sizes": [256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], "max_capture_size": 256}, use_cached_outputs=True,
WARNING 06-19 14:28:14 [multiproc_worker_utils.py:306] Reducing Torch parallelism from 20 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 06-19 14:28:14 [rocm.py:192] None is not supported in AMD GPUs.
INFO 06-19 14:28:14 [rocm.py:193] Using ROCmFlashAttention backend.
INFO 06-19 14:28:16 [init.py:248] Automatically detected platform rocm.
(VllmWorkerProcess pid=303) INFO 06-19 14:28:20 [multiproc_worker_utils.py:225] Worker ready; awaiting tasks
(VllmWorkerProcess pid=303) INFO 06-19 14:28:20 [init.py:30] Available plugins for group vllm.general_plugins:
(VllmWorkerProcess pid=303) INFO 06-19 14:28:20 [init.py:32] name=lora_filesystem_resolver, value=vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
(VllmWorkerProcess pid=303) INFO 06-19 14:28:20 [init.py:34] all available plugins for group vllm.general_plugins will be loaded.
(VllmWorkerProcess pid=303) INFO 06-19 14:28:20 [init.py:36] set environment variable VLLM_PLUGINS to control which plugins to load.
(VllmWorkerProcess pid=303) INFO 06-19 14:28:20 [init.py:44] plugin lora_filesystem_resolver loaded.
(VllmWorkerProcess pid=303) INFO 06-19 14:28:20 [rocm.py:192] None is not supported in AMD GPUs.
(VllmWorkerProcess pid=303) INFO 06-19 14:28:20 [rocm.py:193] Using ROCmFlashAttention backend.
(VllmWorkerProcess pid=303) INFO 06-19 14:28:21 [utils.py:1220] Found nccl from library librccl.so.1
(VllmWorkerProcess pid=303) INFO 06-19 14:28:21 [pynccl.py:69] vLLM is using nccl==2.21.5
INFO 06-19 14:28:21 [utils.py:1220] Found nccl from library librccl.so.1
INFO 06-19 14:28:21 [pynccl.py:69] vLLM is using nccl==2.21.5
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] Exception in worker VllmWorkerProcess while processing method init_device.
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] Traceback (most recent call last):
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/multiproc_worker_utils.py", line 232, in _run_worker_process
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] output = run_method(worker, method, args, kwargs)
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2769, in run_method
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] return func(*args, **kwargs)
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 604, in init_device
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] self.worker.init_device() # type: ignore
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 218, in init_device
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] init_worker_distributed_environment(self.vllm_config, self.rank,
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 560, in init_worker_distributed_environment
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1102, in ensure_model_parallel_initialized
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] initialize_model_parallel(tensor_model_parallel_size,
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1040, in initialize_model_parallel
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] _TP = init_model_parallel_group(group_ranks,
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 805, in init_model_parallel_group
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] return GroupCoordinator(
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 253, in init
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] self.device_communicator = device_comm_cls(
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 46, in init
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] self.pynccl_comm = PyNcclCommunicator(
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 105, in init
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] self.all_reduce(data)
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 126, in all_reduce
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] self.nccl.ncclAllReduce(buffer_type(in_tensor.data_ptr()),
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 290, in ncclAllReduce
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] self.NCCL_CHECK(self._funcs["ncclAllReduce"](sendbuff, recvbuff, count,
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 256, in NCCL_CHECK
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] raise RuntimeError(f"NCCL error: {error_str}")
(VllmWorkerProcess pid=303) ERROR 06-19 14:28:21 [multiproc_worker_utils.py:238] RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)

"

[schung-amd]

Thanks for the quick response, I'll take a look.

[schung-amd]

Do you have ROCm installed on the host? Also, did you re-download the model for that test?