[release] 0.9.0rc1 release checklist

[wangxiyuan]

This issue track the workflow for the next release 0.9.0rc1

PR merged

Need merge first:
#1016

• [CI] Add accuracy test for Qwen2.5-VL-3B-Instruct #766
• [Bugfix][Worker] Clear NPU memory between test profiling #989
• [CI] Re-enable sleep mode test and skip failure breaking CI #990
• [CI] remove old quantization model #1003
• [Patch] Remove spec_decode.metrics patch #1016
• [Misc] Refactor additional_config #1029
• [ModelRunner] Support embedding inputs #916

Need review and merge:

• dense and multimodal

• Add qwen2.5 vl multimodal feature for vllm-ascend v1 #736
• [V1][Structured Output] Enable Speculative Decoding with Structured Outputs #751

• DeepSeek:

• optimize the funtion of computing topk and topp in sampler. #970
• MLA layer eliminates redundant index operators #993
• feat: support data parallel for deepseek #1012
• [MTP][V1] Adapt mtp with graph mode in v1. #1023
• [Bugfix] Fix deepseek percision issue and add acc ci for it #905
• [Performance] Add EPLB expert map import capabilities #919
• [perf]: support dual-batch overlap(dbo) for deepseek #941

• RL:

• [Kernel] Remove cumsum in groupedmatmul #987
• Support multistream of shared experts in FusedMoE #997
• [CI]Moe alltoall communication optimization #1067
• [ModelRunner]Add profile execute duration observation  #1013

Pending (will NOT be included in 0.9.0rc1):

• add custom ascendc kernel vocabparallelembedding #796

• [perf] Improve Prefill Performance by Optimizing Alltoall Communication #978

• [ModelRunner][MultiModal] Automatically cast multi-modal input dtype #1002

• [V1][eagle3] Support eagle3 proposer for v1 #1032

• [Performance] [flash_communication_v1] DeepSeek communication optimization on A2 (reduce_scatter + all_gather) #1034

• [Draft]mooncake connector v1 #1036

• kvcache nz #1054

• Revert the modifications of cache engine for npu graph mode #875

• Disaggregate prefill for kv cache register style #950

• [Feature] Synchronize vLLM mrope modifications, support Qwen2.5-OMni-7B Thinker. #973 need V1 support

• [Platform] Add support for Ascend 310P #914 need torch npu public new version

requirement

• PTA + CANN upgrade

Functional Test (V1)

• Qwen3/Qwen2.5: aclgraph + Qwen2.5/Qwen3 @MengqingCao

  • [Bug]: Failed to complete vllm benchmark after enable VLLM_USE_V1=1 due to gather_v3 error #1038
  • [Bug][V1]: Qwen/Qwen2.5-7B-Instruct accuracy ceval-valid failed #1043

• Qwen2.5 VL: eager mode @shen-shanshan @ChenTaoyu-SJTU

  • [Bug][V1]: Failed to start Qwen/Qwen2.5-VL-7B-Instruct accuracy serve #1044

• DeepSeek: torchair + deepseek @zzzzwwjj

• Quantization (w8a8) + Qwen2.5/Qwen3/DeepSeek @22dimensions

  • Modelscope download
    • Refresh readme
    • https://modelscope.cn/models/vllm-ascend/DeepSeek-R1-0528-W8A8
    • https://modelscope.cn/models/vllm-ascend/Qwen3-8B-W8A8
    • https://modelscope.cn/models/vllm-ascend/Qwen2.5-0.5B-Instruct-W8A8
  • w8a8 e2e test refresh to new test
  • doc: User Guide - quantization

• spec decode + mtp @mengwei805

• performance @Potabk

  • V0 Qwen3
  • V0 Qwen2.5
  • V0 Qwen2.5 VL
  • V1 Qwen3 need bugfix
  • V1 Qwen2.5 need bugfix
  • V1 Qwen2.5 VL

• accuracy @zhangxinyuehfad

  • [CI] Make accuarcy CI and report work #1078

• DP @MengqingCao @yiz-liu

  • dp+enger
  • dp+torchair

• EP @wangxiyuan

  • EPLB

• PD @wangxiyuan

  • 1P1D E2E - simple connector
  • 1P1D E2E - pyhccl

• RL related @leo-pony

  • sleep mode
  • VL pad @shen-shanshan

• (V0) Input embeding @Potabk

• (V1) Structed ouput @shen-shanshan

• (V1) AscendScheduler @MengqingCao

• (V1) Scheduler CP/APC @wangxiyuan

Documentation

• addintion config
• environment
• release note
• graph mode

Release

• tag、binary、image、pypi

[henryxuxu0716]

#1067
#987
#997
#1013

Need review and merge included in 0.9.0rc1

[Yikun]

@henryxuxu0716 updated, thanks for feedback

[wangtiance]

so xPyD disaggregation will not be supported in 0.9.0rc1?

[Yikun]

@wangtiance Yes, we are still working on xPyD. It will ready in Q2.

[shen-shanshan]

Qwen2.5-VL: eager mode test (V1)

Functional Test

Test Qwen2.5-VL on V1 using:

# Launch the server
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
--dtype bfloat16 \
--max_model_len 32768 \
--max-num-batched-tokens 32768 \
--enforce-eager

# Curl the server
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
    "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/qwen.png"}},
        {"type": "text", "text": "What is the text in the illustrate?"}
    ]}
    ]
    }'

Result:

{"id":"chatcmpl-60e4a8a6b9574a7a9beb969fecf18993","object":"chat.completion","created":1749109761,"model":"Qwen/Qwen2.5-VL-7B-Instruct","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"The text in the illustration reads \"TONGYI Qwen.\"","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":73,"total_tokens":88,"completion_tokens":15,"prompt_tokens_details":null},"prompt_logprobs":null,"kv_transfer_params":null}

Benchmark

Benchmark Qwen2.5-VL on V1 using:

# Launch the server
python -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-VL-7B-Instruct \
--tensor-parallel-size 1 \
--swap-space 16 \
--disable-log-stats \
--disable-log-requests \
--load-format dummy \
--enforce-eager

# Run benchmark
cd /vllm-workspace/vllm/benchmarks
python benchmark_serving.py \
--model Qwen/Qwen2.5-VL-7B-Instruct \
--dataset-name random \
--random-input-len 200 \
--num-prompts 200 \
--request-rate 1 \
--save-result --result-dir ./

Result:

============ Serving Benchmark Result ============
Successful requests:                     200       
Benchmark duration (s):                  191.23    
Total input tokens:                      40000     
Total generated tokens:                  25600     
Request throughput (req/s):              1.05      
Output token throughput (tok/s):         133.87    
Total Token throughput (tok/s):          343.04    
---------------Time to First Token----------------
Mean TTFT (ms):                          84.26     
Median TTFT (ms):                        84.03     
P99 TTFT (ms):                           110.59    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          50.63     
Median TPOT (ms):                        50.51     
P99 TPOT (ms):                           53.83     
---------------Inter-token Latency----------------
Mean ITL (ms):                           50.63     
Median ITL (ms):                         50.15     
P99 ITL (ms):                            56.58     
==================================================

Accuracy Test

Test Qwen2.5-VL on V1 using:

export MODEL_ARGS='pretrained=Qwen/Qwen2.5-VL-7B-Instruct,max_model_len=8192,dtype=auto,tensor_parallel_size=1,max_images=2,enforce_eager=True'

lm_eval \
--model vllm-vlm \
--model_args $MODEL_ARGS \
--tasks mmmu_val \
--apply_chat_template \
--fewshot_as_multiturn \
--batch_size 1

Result:

|   Task   | Filter | n-shot | Metric | Value  |  Stderr  |
| -------- | -----: | -----: | ------ | -----: | -------: |
| mmmu_val |   none |      0 | acc    | 0.5144 | ± 0.0162 |

[shen-shanshan]

Need merge:

• [Misc] Add ACL_OP_INIT_MODE env var and set default to 1 #597
• [Bugfix][Worker] Clear NPU memory between test profiling #989
• [Patch] Remove spec_decode.metrics patch #1016

[Potabk]

Need merge:
#1104
#1084
#916

[MengqingCao]

AscendScheduler test pass with the following test:

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]

# Create a sampling params object.
sampling_params = SamplingParams(max_tokens=100, temperature=0.8)
# Create an LLM.
llm = LLM(model="deepseek-ai/DeepSeek-V2-Lite-Chat",
    additional_config={
        'ascend_scheduler_config': {},
    },
    trust_remote_code=True,
    max_model_len=102,
    enforce_eager=True,
    enable_prefix_caching=True,
    enable_chunked_prefill=True
)

# Generate texts from the prompts.
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

[MengqingCao]

Need merge:

• [Bugfix][DP] Add with_prefill_across_dp to AscendMetadata to fix dp #1094

[Yikun]

Run Accuracy test here: https://github.com/vllm-project/vllm-ascend/actions/runs/15512907509?pr=1078

🎯 Qwen2.5-7B-Instruct Accuracy Test (V0)

vLLM version: vLLM: 0.9.0, vLLM Ascend: refs/pull/1078/merge

Software Environment: CANN: 8.1.RC1, PyTorch: 2.5.1, torch-npu: 2.5.1

Hardware Environment: Atlas A2 Series

Datasets: ceval-valid,gsm8k

Command:

export MODEL_ARGS='pretrained=Qwen/Qwen2.5-7B-Instruct, max_model_len=4096,dtype=auto,tensor_parallel_size=2,gpu_memory_utilization=0.6'
lm_eval --model vllm --model_args $MODEL_ARGS --tasks ceval-valid,gsm8k \ 
--apply_chat_template --fewshot_as_multiturn --num_fewshot 5 --batch_size 1

 

Task	Filter	n-shot	Metric	Value	Stderr
ceval-valid	none	5	acc	↑ 0.8001	± 0.0105
gsm8k	flexible-extract	5	exact_match	↑ 0.7278	± 0.0123

🎯 Qwen2.5-7B-Instruct Accuracy Test (V1)

vLLM version: vLLM: 0.9.0, vLLM Ascend: refs/pull/1078/merge

Software Environment: CANN: 8.1.RC1, PyTorch: 2.5.1, torch-npu: 2.5.1

Hardware Environment: Atlas A2 Series

Datasets: ceval-valid,gsm8k

Command:

export MODEL_ARGS='pretrained=Qwen/Qwen2.5-7B-Instruct, max_model_len=4096,dtype=auto,tensor_parallel_size=2,gpu_memory_utilization=0.6'
lm_eval --model vllm --model_args $MODEL_ARGS --tasks ceval-valid,gsm8k \ 
--apply_chat_template --fewshot_as_multiturn --num_fewshot 5 --batch_size 1

 

Task	Filter	n-shot	Metric	Value	Stderr
ceval-valid	none	5	acc	↑ 0.2303	± 0.0115
gsm8k	flexible-extract	5	exact_match	↑ 0.7248	± 0.0123

🎯 Qwen2.5-VL-7B-Instruct Accuracy Test (V0)

vLLM version: vLLM: 0.9.0, vLLM Ascend: refs/pull/1078/merge

Software Environment: CANN: 8.1.RC1, PyTorch: 2.5.1, torch-npu: 2.5.1

Hardware Environment: Atlas A2 Series

Datasets: mmmu_val

Command:

export MODEL_ARGS='pretrained=Qwen/Qwen2.5-VL-7B-Instruct, max_model_len=8192,dtype=auto,tensor_parallel_size=4,max_images=2'
lm_eval --model vllm-vlm --model_args $MODEL_ARGS --tasks mmmu_val \ 
--apply_chat_template --fewshot_as_multiturn  --batch_size 1

 

Task	Filter	n-shot	Metric	Value	Stderr
mmmu_val	none	0	acc	↑ 0.5178	± 0.0162

[Yikun]

Run Qwen Performance test compare with v0.7.3:

============ Serving Benchmark Result ============
Successful requests:                     200
Benchmark duration (s):                  187.35
Total input tokens:                      40000
Total generated tokens:                  25600
Request throughput (req/s):              1.07
Output token throughput (tok/s):         136.64
Total Token throughput (tok/s):          350.14
---------------Time to First Token----------------
Mean TTFT (ms):                          49.43
Median TTFT (ms):                        48.77
P99 TTFT (ms):                           66.11
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          22.03
Median TPOT (ms):                        22.64
P99 TPOT (ms):                           26.12
---------------Inter-token Latency----------------
Mean ITL (ms):                           22.03
Median ITL (ms):                         25.11
P99 ITL (ms):                            32.06
==================================================

Base Image		TTFT	Improvement	TPOT	Improvement	ITL	Improvement
vllm-ascend:v0.7.3	vLLM Ascend v0.7.3 (Baseline)	86.58	0.00%	32.57	0.00%	32.57	0.00%
vllm-ascend:v0.7.3	vLLM Ascend v0.7.3 + MindIE Turbo	64.45	25.56%	21.94	32.64%	21.94	32.64%
vllm-ascend:main(0608, v0.9.0 vllm)	vLLM Ascend v0.9.0rc1.dev0608 Enable V1 Round1	50.96	41.14%	22.38	31.29%	24.76	23.98%
vllm-ascend:main(0608, v0.9.0 vllm)	vLLM Ascend v0.9.0rc1.dev0608 Enable V1 Round2	45.24	47.75%	22.66	30.43%	25.44	21.89%
vllm-ascend:main(0608, v0.9.0 vllm)	vLLM Ascend v0.9.0rc1.dev0608 Enable V1 Round3	48.7	43.75%	22.64	30.49%	25.11	22.90%
vllm-ascend:main(0608, v0.9.0 vllm)	vLLM Ascend v0.9.0rc1.dev0608 V0	82.91	4.24%	32.41	0.49%	31.26	4.02%

[Yikun]

1P1D CI failed, not sure a real issue here or not:
https://github.com/vllm-project/vllm-ascend/actions/runs/15513454998/job/43677261860

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/e2e/pd_disaggreate/test_pd_e2e.py:77: in start_and_test_pipeline
    wait_for_port(DECODE_PORT, 600)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

port = 8002, timeout = 600

    def wait_for_port(port, timeout=30):
        import socket
        start = time.time()
        while time.time() - start < timeout:
            with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
                if sock.connect_ex(("127.0.0.1", port)) == 0:
                    return True
            time.sleep(1)
>       raise TimeoutError(f"Port {port} not ready after {timeout}s")
E       TimeoutError: Port 8002 not ready after 600s

tests/e2e/pd_disaggreate/test_pd_e2e.py:68: TimeoutError
=========================== short test summary info ============================
FAILED tests/e2e/pd_disaggreate/test_pd_e2e.py::test_disaggregated_pd_pipeline - TimeoutError: Port 8002 not ready after 600s
======================== 1 failed in 602.92s (0:10:02) =========================