[Bug]: [V1] DeepSeek MTP is broken since 0.9.0

[cjackal]

Your current environment

The output of python collect_env.py

==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.7.0+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : Python version : 3.10.12 (main, May 27 2025, 17:12:29) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-5.14.0-284.115.1.el9_2.x86_64-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : Could not collect
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration :
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3
GPU 4: NVIDIA H100 80GB HBM3
GPU 5: NVIDIA H100 80GB HBM3
GPU 6: NVIDIA H100 80GB HBM3
GPU 7: NVIDIA H100 80GB HBM3

Nvidia driver version        : 570.148.08
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        52 bits physical, 57 bits virtual
Byte Order:                           Little Endian
CPU(s):                               64
On-line CPU(s) list:                  0-63
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) Gold 6430
CPU family:                           6
Model:                                143
Thread(s) per core:                   1
Core(s) per socket:                   32
Socket(s):                            2
Stepping:                             8
CPU max MHz:                          3400.0000
CPU min MHz:                          800.0000
BogoMIPS:                             4200.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
L1d cache:                            3 MiB (64 instances)
L1i cache:                            2 MiB (64 instances)
L2 cache:                             128 MiB (64 instances)
L3 cache:                             120 MiB (2 instances)
NUMA node(s):                         8
NUMA node0 CPU(s):                    0-7
NUMA node1 CPU(s):                    8-15
NUMA node2 CPU(s):                    16-23
NUMA node3 CPU(s):                    24-31
NUMA node4 CPU(s):                    32-39
NUMA node5 CPU(s):                    40-47
NUMA node6 CPU(s):                    48-55
NUMA node7 CPU(s):                    56-63
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.2.6.post1+cu128torch2.7
[pip3] mypy-extensions==1.0.0
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.3.14
[pip3] nvidia-cuda-cupti-cu12==12.8.57
[pip3] nvidia-cuda-nvrtc-cu12==12.8.61
[pip3] nvidia-cuda-runtime-cu12==12.8.57
[pip3] nvidia-cudnn-cu12==9.7.1.26
[pip3] nvidia-cufft-cu12==11.3.3.41
[pip3] nvidia-cufile-cu12==1.13.0.11
[pip3] nvidia-curand-cu12==10.3.9.55
[pip3] nvidia-cusolver-cu12==11.7.2.55
[pip3] nvidia-cusparse-cu12==12.5.7.53
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.8.61
[pip3] nvidia-nvtx-cu12==12.8.55
[pip3] pyzmq==26.4.0
[pip3] torch==2.7.0+cu128
[pip3] torchaudio==2.7.0+cu128
[pip3] torchvision==0.22.0+cu128
[pip3] transformers==4.52.4
[pip3] triton==3.3.0
[conda] No relevant packages

==============================
         vLLM Info
==============================
ROCM Version : Could not collect
Neuron SDK Version : N/A
vLLM Version : 0.9.2.dev223+gee5ad8d2c (git sha: ee5ad8d2c)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    CPU Affinity    NUMA Affinity    GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     SYS     SYS     SYS     SYS     0-7     0    N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    PXB     SYS     SYS     SYS     SYS     0-7     0    N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    SYS     PXB     NODE    SYS     SYS     16-23   2    N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    SYS     PIX     NODE    SYS     SYS     16-23   2    N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     PXB     SYS     32-39   4    N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     PIX     SYS     32-39   4    N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     PXB     48-55   6    N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     PIX     48-55   6    N/A
NIC0    PIX     PXB     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS     SYS     SYS     SYS
NIC1    SYS     SYS     PXB     PIX     SYS     SYS     SYS     SYS     SYS      X      NODE    SYS     SYS
NIC2    SYS     SYS     NODE    NODE    SYS     SYS     SYS     SYS     SYS     NODE     X      SYS     SYS
NIC3    SYS     SYS     SYS     SYS     PXB     PIX     SYS     SYS     SYS     SYS     SYS      X      SYS
NIC4    SYS     SYS     SYS     SYS     SYS     SYS     PXB     PIX     SYS     SYS     SYS     SYS      X

Legend:

  X = Self
  SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX = Connection traversing at most a single PCIe bridge
  NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=/var/run/nvidia-container-devices
NCCL_DEBUG=INFO
NCCL_IB_HCA=mlx5
LD_LIBRARY_PATH=/usr/local/cuda/lib64
VLLM_HOST_IP=172.16.14.32
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
VLLM_WORKER_MULTIPROC_METHOD=spawn
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

DeepSeek MTP speculative decoder, which is available on V1 since #18435, is broken on current main branch. I haven't bisect thoroughly, but it was broken at least since v0.9.0.

Launch options:

vllm serve /app/model/DEEPSEEK-R1/ \
  --served-model-name deepseek-ai/deepseek-r1 \
  --gpu-memory-utilization 0.95 \
  --tensor-parallel-size 16 \
  --max-model-len 65536 \
  --max-num-batched-tokens 8192 \
  --reasoning-parser deepseek_r1 \
  --speculative-config '{"method":"deepseek_mtp","num_speculative_tokeens":1}'

On current main(0.9.2.dev223+gee5ad8d2c), the following error is raised:

...
Loading safetensors checkpoint shards: 94% Completed | 154/163 [00:25<00:00, 46.74it/s]
(RayWorkerWrapper pid=8789) INFO 06-24 14:22:13 [default_loader.py:272] Loading weights took 27.56 seconds [repeated 8x across cluster]
(RayWorkerWrapper pid=8813) INFO 06-24 14:21:45 [gpu_model_runner.py:1721] Loading drafter model... [repeated 7x across cluster]
(RayWorkerWrapper pid=8813) WARNING 06-24 14:21:45 [utils.py:303] The model class DeepSeekMTP has not defined `packed_modules_mapping`, this may lead to incorrect mapping of quantized or ignored modules [repeated 7x across cluster]
(RayWorkerWrapper pid=8813) WARNING 06-24 14:21:45 [config.py:4734] `torch.compile` is turned on, but the model /app/model/DEEPSEEK-R1/ does not support it. Please open an issue on GitHub if you want it to be supported. [repeated 7x across cluster]
Loading safetensors checkpoint shards: 98% Completed | 160/163 [00:27<00:00, 7.77it/s]
Loading safetensors checkpoint shards: 100% Completed | 163/163 [00:27<00:00, 5.89it/s]
(RayWorkerWrapper pid=8804) ERROR 06-24 14:22:13 [core.py:519] EngineCore failed to start.
ERROR 06-24 14:22:13 [core.py:519] Traceback (most recent call last):
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 510, in run_engine_core
ERROR 06-24 14:22:13 [core.py:519] engine_core = EngineCoreProc(*args, **kwargs)
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 394, in __init__
ERROR 06-24 14:22:13 [core.py:519] super().__init__(vllm_config, executor_class, log_stats,
ERROR 06-24 14:22:13 [core.py:519] File "/home/jovyan/git/vllm-serving/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 75, in __init__
ERROR 06-24 14:22:13 [core.py:519] self.model_executor = executor_class(vllm_config)
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 287, in __init__
ERROR 06-24 14:22:13 [core.py:519] super().__init__(*args, **kwargs)
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 53, in __init__
ERROR 06-24 14:22:13 [core.py:519] self._init_executor()
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/vllm/executor/ray_distributed_executor.py", line 115, in _init_executor
ERROR 06-24 14:22:13 [core.py:519] self._init_workers_ray(placement_group)
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/vllm/executor/ray_distributed_executor.py", line 397, in _init_workers_ray
ERROR 06-24 14:22:13 [core.py:519] self._run_workers("load_model",
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/vllm/executor/ray_distributed_executor.py", line 522, in _run_workers
ERROR 06-24 14:22:13 [core.py:519] ray_worker_outputs = ray.get(ray_worker_outputs)
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
ERROR 06-24 14:22:13 [core.py:519] return fn(*args, **kwargs)
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
ERROR 06-24 14:22:13 [core.py:519] return func(*args, **kwargs)
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/ray/_private/worker.py", line 2822, in get
ERROR 06-24 14:22:13 [core.py:519] values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/ray/_private/worker.py", line 930, in get_objects
ERROR 06-24 14:22:13 [core.py:519] raise value.as_instanceof_cause()
ERROR 06-24 14:22:13 [core.py:519] ray.exceptions.RayTaskError(AttributeError): ray::RayWorkerWrapper.execute_method() (pid=8801, ip=172.16.14.32, actor_id=c5f06cc1ae932fbe846f5f3702000000, repr=<vllm.executor.ray_utils.RayWorkerWrapper object at 0x7f27115c9e40>)
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 623, in execute_method
ERROR 06-24 14:22:13 [core.py:519] raise e
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 614, in execute_method
ERROR 06-24 14:22:13 [core.py:519] return run_method(self, method, args, kwargs)
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/vllm/utils.py", line 2687, in run_method
ERROR 06-24 14:22:13 [core.py:519] return func(*args, **kwargs)
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 185, in load_model
ERROR 06-24 14:22:13 [core.py:519] self.model_runner.load_model()
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1722, in load_model
ERROR 06-24 14:22:13 [core.py:519] self.drafter.load_model(self.model)
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/vllm/v1/spec_decode/eagle.py", line 345, in load_model
ERROR 06-24 14:22:13 [core.py:519] and self.model.model.embed_tokens.weight.shape \
ERROR 06-24 14:22:13 [core.py:519] File "/app/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1940, in __getattr__
ERROR 06-24 14:22:13 [core.py:519] raise AttributeError(
ERROR 06-24 14:22:13 [core.py:519] AttributeError: 'DeepSeekMultiTokenPredictor' object has no attribute 'embed_tokens'

On v0.9.0 where the multi KV cache group was yet to be introduced, the server frontend becomes ready, but it raises the following error if a request is consumed:

...
INFO 06-24 15:14:37 [async_llm.py:270] Added request chatcmpl-78a6a56e274f431e87922271e5d0c5c1.
INFO 06-24 15:14:37 [ray_distributed_executor.py:569] RAY_CGRAPH_get_timeout is set to 300
INFO 06-24 15:14:37 [ray_distributed_executor.py:571] VLLM_USE_RAY_COMPILED_DAG_CHANNEL_TYPE = auto
INFO 06-24 15:14:37 [ray_distributed_executor.py:573] VLLM_USE_RAY_COMPILED_DAG_OVERLAP_COMM = False
ERROR 06-24 15:14:38 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.9.2.dev223+gee5ad8d2c) with config: model='/app/model/DEEPSEEK-R1/', speculative_config=SpeculativeConfig(method='deepseek_mtp', model='/app/model/DEEPSEEK-R1/', num_spec_tokens=1), tokenizer='/app/model/DEEPSEEK-R1/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=16, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend='deepseek_r1'), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=deepseek-ai/deepseek-r1, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null},
ERROR 06-24 15:14:38 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-78a6a56e274f431e87922271e5d0c5c1,prompt_token_ids_len=21,mm_inputs=[],mm_hashes=[],mm_positions=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=131051, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None),block_ids=([1],),num_computed_tokens=0,lora_request=None)], scheduled_cached_reqs=[], num_scheduled_tokens={chatcmpl-78a6a56e274f431e87922271e5d0c5c1: 21}, total_num_scheduled_tokens=21, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[1], finished_req_ids=[], free_encoder_input_ids=[], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null)
ERROR 06-24 15:14:38 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, kv_cache_usage=0.00029766334275938, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=21, hits=0), spec_decoding_stats=None, num_corrupted_reqs=0)
ERROR 06-24 15:14:38 [core.py:521] EngineCore encountered a fatal error.
ERROR 06-24 15:14:38 [core.py:521] Traceback (most recent call last):
ERROR 06-24 15:14:38 [core.py:521] File "/app/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 512, in run_engine_core
ERROR 06-24 15:14:38 [core.py:521] engine_core.run_busy_loop()
ERROR 06-24 15:14:38 [core.py:521] File "/app/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 539, in run_busy_loop
ERROR 06-24 15:14:38 [core.py:521] self._process_engine_step()
ERROR 06-24 15:14:38 [core.py:521] File "/app/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 564, in _process_engine_step
ERROR 06-24 15:14:38 [core.py:521] outputs, model_executed = self.step_fn()
ERROR 06-24 15:14:38 [core.py:521] File "/app/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 235, in step
ERROR 06-24 15:14:38 [core.py:521] model_output = self.execute_model(scheduler_output)
ERROR 06-24 15:14:38 [core.py:521] File "/app/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 221, in execute_model
ERROR 06-24 15:14:38 [core.py:521] raise err
ERROR 06-24 15:14:38 [core.py:521] File "/app/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 212, in execute_model
ERROR 06-24 15:14:38 [core.py:521] return self.model_executor.execute_model(scheduler_output)
ERROR 06-24 15:14:38 [core.py:521] File "/app/.venv/lib/python3.10/site-packages/vllm/v1/executor/ray_distributed_executor.py", line 58, in execute_model
ERROR 06-24 15:14:38 [core.py:521] return refs[0].get()
ERROR 06-24 15:14:38 [core.py:521] File "/app/.venv/lib/python3.10/site-packages/ray/experimental/compiled_dag_ref.py", line 150, in get
ERROR 06-24 15:14:38 [core.py:521] return _process_return_vals(return_vals, True)
ERROR 06-24 15:14:38 [core.py:521] File "/app/.venv/lib/python3.10/site-packages/ray/experimental/compiled_dag_ref.py", line 27, in _process_return_vals
ERROR 06-24 15:14:38 [core.py:521] raise val.as_instanceof_cause()
ERROR 06-24 15:14:38 [core.py:521] ray.exceptions.RayTaskError(AttributeError): ray::RayWorkerWrapper.__ray_call__() (pid=8817, ip=172.16.14.32)
ERROR 06-24 15:14:38 [core.py:521] File "/app/.venv/lib/python3.10/site-packages/vllm/executor/ray_utils.py", line 135, in execute_model_ray
ERROR 06-24 15:14:38 [core.py:521] output = self.worker.model_runner.execute_model(
ERROR 06-24 15:14:38 [core.py:521] File "/app/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 06-24 15:14:38 [core.py:521] return func(*args, **kwargs)
ERROR 06-24 15:14:38 [core.py:521] File "/app/.venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1583, in execute_model
ERROR 06-24 15:14:38 [core.py:521] draft_token_ids = self.drafter.propose(
ERROR 06-24 15:14:38 [core.py:521] File "/app/.venv/lib/python3.10/site-packages/vllm/v1/spec_decode/eagle.py", line 151, in propose
ERROR 06-24 15:14:38 [core.py:521] attn_metadata = self.runner.attn_metadata_builder.build(
ERROR 06-24 15:14:38 [core.py:521] AttributeError: 'GPUModelRunner' object has no attribute 'attn_metadata_builder'
INFO 06-24 15:14:38 [ray_distributed_executor.py:128] Shutting down Ray distributed executor. If you see error log from logging.cc regarding SIGTERM received, please ignore because this is the expected termination process in Ray.
2025-06-24 15:14:38,236 INFO compiled_dag_node.py:2173 -- Tearing down compiled DAG

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[josephrocca]

Thank you for this fix! I also noticed the attn_metadata_builder error a few days ago but didn't know how to fix it.

Testing latest commit on main as of your fix being merged, and getting this error for --speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":1}' - wondering whether I'm doing something silly here?

WorkerProc hit an exception.                                                                                       
Traceback (most recent call last):                                                                                 
  File "/root/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop                          
    output = func(*args, **kwargs)                                                                                 
             ^^^^^^^^^^^^^^^^^^^^^                                                                                 
  File "/opt/conda/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context         
    return func(*args, **kwargs)                                                                                   
           ^^^^^^^^^^^^^^^^^^^^^                                                                                   
  File "/root/vllm/vllm/v1/worker/gpu_worker.py", line 303, in execute_model                                       
    output = self.model_runner.execute_model(scheduler_output,                                                     
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                     
  File "/opt/conda/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context         
    return func(*args, **kwargs)                                                                                   
           ^^^^^^^^^^^^^^^^^^^^^                                                                                   
  File "/root/vllm/vllm/v1/worker/gpu_model_runner.py", line 1527, in execute_model                                
    next_token_id = req_state.get_token_id(seq_len)                                                                
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/vllm/vllm/v1/worker/gpu_input_batch.py", line 56, in get_token_id
    return self.output_token_ids[idx - self.num_prompt_tokens]
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: list index out of range

My full command is:

python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 3000 --max-model-len 8192 --max-seq-len-to-capture 8192 --enable-chunked-prefill --enable-prefix-caching --trust-remote-code --disable-log-requests --tensor-parallel-size 8 --gpu-memory-utilization 0.95 --served-model-name deepseek-chat --model RedHatAI/DeepSeek-R1-0528-quantized.w4a16 --compilation-config '{"full_cuda_graph": true}' --speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":1}'

Then I sent 200 concurrent requests at it, and got the above error.

I considered maybe it's related to your comment on the pull request? (completely random guess)

I need better understanding on multiple KV cache group scenario, so leave it as a future work.

[cjackal]

Thank you for this fix! I also noticed the attn_metadata_builder error a few days ago but didn't know how to fix it.

Testing latest commit on main as of your fix being merged, and getting this error for --speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":1}' - wondering whether I'm doing something silly here?

WorkerProc hit an exception.                                                                                       
Traceback (most recent call last):                                                                                 
  File "/root/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop                          
    output = func(*args, **kwargs)                                                                                 
             ^^^^^^^^^^^^^^^^^^^^^                                                                                 
  File "/opt/conda/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context         
    return func(*args, **kwargs)                                                                                   
           ^^^^^^^^^^^^^^^^^^^^^                                                                                   
  File "/root/vllm/vllm/v1/worker/gpu_worker.py", line 303, in execute_model                                       
    output = self.model_runner.execute_model(scheduler_output,                                                     
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                     
  File "/opt/conda/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context         
    return func(*args, **kwargs)                                                                                   
           ^^^^^^^^^^^^^^^^^^^^^                                                                                   
  File "/root/vllm/vllm/v1/worker/gpu_model_runner.py", line 1527, in execute_model                                
    next_token_id = req_state.get_token_id(seq_len)                                                                
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/vllm/vllm/v1/worker/gpu_input_batch.py", line 56, in get_token_id
    return self.output_token_ids[idx - self.num_prompt_tokens]
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

My full command is:

python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 3000 --max-model-len 8192 --max-seq-len-to-capture 8192 --enable-chunked-prefill --enable-prefix-caching --trust-remote-code --disable-log-requests --tensor-parallel-size 8 --gpu-memory-utilization 0.95 --served-model-name deepseek-chat --model RedHatAI/DeepSeek-R1-0528-quantized.w4a16 --compilation-config '{"full_cuda_graph": true}' --speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":1}'

Then I sent 200 concurrent requests at it, and got the above error.

I considered maybe it's related to your comment on the pull request? (completely random guess)

I need better understanding on multiple KV cache group scenario, so leave it as a future work.

I haven't checked yet but it is possible that the quantized checkpoint that you are using may not properly configure the MTP layer; it is often the case that MTP layer is dropped in quantization process.

[josephrocca]

@cjackal Strangely, I just tried the quantized checkpoint again, this time with --gpu-memory-utilization set to 0.90 (previously I tried with 0.95) and this time I got an illegal memory access error:

Traceback (most recent call last):                                                                                 
  File "/root/vllm/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop                          
    output = func(*args, **kwargs)                                                                                 
             ^^^^^^^^^^^^^^^^^^^^^                                                                                 
  File "/opt/conda/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context         
    return func(*args, **kwargs)                                                                                   
           ^^^^^^^^^^^^^^^^^^^^^                                                                                   
  File "/root/vllm/vllm/v1/worker/gpu_worker.py", line 303, in execute_model                                       
    output = self.model_runner.execute_model(scheduler_output,                                                     
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                     
  File "/opt/conda/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context         
    return func(*args, **kwargs)                                                                                   
           ^^^^^^^^^^^^^^^^^^^^^                                                                                   
  File "/root/vllm/vllm/v1/worker/gpu_model_runner.py", line 1476, in execute_model                                
    valid_sampled_token_ids = self.rejection_sampler.parse_output(                                                 
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                 
  File "/root/vllm/vllm/v1/sample/rejection_sampler.py", line 124, in parse_output                                 
    output_token_ids_np = output_token_ids.cpu().numpy()                                                           
                          ^^^^^^^^^^^^^^^^^^^^^^                                                                   
RuntimeError: CUDA error: an illegal memory access was encountered

Here's my full command for that test:

python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 3000 --max-model-len 8192 --max-seq-len-to-capture 8192 --enable-chunked-prefill --enable-prefix-caching --trust-remote-code --disable-log-requests --tensor-parallel-size 8 --gpu-memory-utilization 0.90 --served-model-name deepseek-chat --model RedHatAI/DeepSeek-R1-0528-quantized.w4a16 --compilation-config '{"full_cuda_graph": true}' --speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":1}'

The same command without --speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":1}' works fine.

I also tried --gpu-memory-utilization 0.95 again just to confirm and I got same the previously mentioned IndexError: list index out of range error (not the illegal memory access one). So it seems like the previous error somehow had to do with VRAM?

In both cases I'm using 9502c38138a03669c4d54225336553db70ad799d (latest commit as of a few hours ago) and I'm building with VLLM_USE_PRECOMPILED=1 pip install --editable .

Maybe I need to use VLLM_USE_PRECOMPILED=0?

Also, I tried switching the model to deepseek-ai/DeepSeek-R1-0528 and it didn't give any errors this time, but all the outputs were nonsense (random letters/numbers). Here was my command:

python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 3000 --max-model-len 8192 --max-seq-len-to-capture 8192 --enable-chunked-prefill --enable-prefix-caching --trust-remote-code --disable-log-requests --tensor-parallel-size 8 --gpu-memory-utilization 0.90 --served-model-name deepseek-chat --model deepseek-ai/DeepSeek-R1-0528 --compilation-config '{"full_cuda_graph": true}' --speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":1}'

And again, the same command without --speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":1}' works fine.

I'll continue testing and let you know if I discover any other info that might be useful.

[cjackal]

@josephrocca As it turns out, there is no MTP layers (61st layer) on your quantized checkpoint, so what I think happened is that the MTP layer is not initialized so it throws OOM-type errors or malfunctions as expected. It would be good to add a validation for the lack of MTP layers and error out with proper error message though, I'll work on it when I have time.

For the original deepseek checkpoint, it is likely that MTP is not compatibld with full CUDAGraph; thanks for the report and let me investigate further.

[cjackal]

Something has changed since the working commit (0.9.2.dev223+gee5ad8d2c plus my PR). I can reproduce the same gibberish on 0.9.2.dev283+ge9fd658af even without the full cudagraph compile option.

After bisecting the commits from what I have worked on for the fix to what the PR has been merged, it seems #19717 breaks MTP.

---

As a reproducer, what I have checked is as follows:

For the commits which merges #19717 and just before it, apply the diffs at #20022 and run the test script.

Then for f59fc60 MTP works normal and the draft acceptance rate in the log is about ~80%. For 015fab8 however, model output is a total gibberish and the draft acceptance rate is almost 0.

---

It seems like #19717 changes the forward pass of MTP module from FlashMLA to FusedMoE where it shouldn't. cc @bnellnm , it would be helpful if you can shed some light on what is going wrong.

[bnellnm]

Something has changed since the working commit (0.9.2.dev223+gee5ad8d2c plus my PR). I can reproduce the same gibberish on 0.9.2.dev283+ge9fd658af even without the full cudagraph compile option.

After bisecting the commits from what I have worked on for the fix to what the PR has been merged, it seems #19717 breaks MTP.

As a reproducer, what I have checked is as follows:

For the commits which merges #19717 and just before it, apply the diffs at #20022 and run the test script.

Then for f59fc60 MTP works normal and the draft acceptance rate in the log is about ~80%. For 015fab8 however, model output is a total gibberish and the draft acceptance rate is almost 0.

It seems like #19717 changes the forward pass of MTP module from FlashMLA to FusedMoE where it shouldn't. cc @bnellnm , it would be helpful if you can shed some light on what is going wrong.

My guess is you are running into the problem fixed by #20152