vLLM Ascend Roadmap Q3 2025

[wangxiyuan]

This is a living document! We are eager to know what do you want for vLLM Ascend in Q3 2025. Any feedback is welcome.

We will complete planning roadmap in mid-July.

---

As a vital component of vLLM, the vLLM Ascend project is dedicated to providing an easy, fast, and cheap LLM serving for everyone on Ascend NPU, and to actively contribute to the enrichment of vLLM.

In 2025 Q2, we have focused on 4 themes: vLLM Ascend for Production, Performance Optimization, Key Features, Ecosystem Connect. In 2025 Q3, we will focus on: V1 Engine fully supports、Quality and Production ready、User / Developer Experience、Competitive for Key Workflow.

1. V1 Engine fully supports

• Stable plugin architecture for hardware platforms
• V1 Engine fully supports and cleanup V0 code path: [Feature]: Enable V1 by default and cleanup V0 code #1620
• Enable CustomOP register: [CustomOP][Refactor] Register CustomOP instead of overwrite forward_oot #1647
• V1 PP supports

2. Quality and Production ready

• Unit test coverage enhancement: [RFC]: Unit test coverage improvement #1298
• Multi-node test

3. User / Developer Experience

• Users doc: [RFC]: Doc enhancement #1248
• Developer Design doc: [RFC]: Doc enhancement #1248
• Distributions
• Perf Dashboard
• Developer Experience
  • vLLM commit hash recording: Record vLLM commit in PR description #1623

4. Competitive for Key Workflow

• Large Scale Serving

  • EPLB
    • Dynamic EPLB: [Feature] Dynamic Expert Load Balance Zero-like-overhead #1391
    • static EPLB: Add static EPLB #1116
  • Qwen series (Qwen3 / Qwen3 MoE) optimization [Perf] Optimize perf of Qwen3 #1245
  • Qwen series (Qwen3 MoE) optimization: [Bugfix] Support Qwen3-MOE on aclgraph mode #1381
  • Disaggregated Prefilling
    • LLMDataDist: Disaggregate prefill for kv cache register style #950
    • Mooncake: [P/D] Mooncake Connector for v1 distributed #1568

• RLHF

  • Performance improvements
  • Parallel support

• Model

  • Qwen/DeepSeek/Qwen VL series
  • Gemma3
  • New model support vLLM Ascend Model Support Priority #1608
    - New trending models like: minimax / hunyuan / ERNIE
  • Quantization support: w4a16/w4a8 for Dense model
  • Quantization support: w4a16/w4a8 for MoE model
  • Model format support: gguf

• Others

  • Atlas 300I series experimental support and perf enhancement: [Performance] Disable JIT and nd2nz to improve performance for Altlas 300I series #1591

---

If any of the items you wanted is not on the roadmap, your suggestion and contribution is strongly welcomed! Please feel free to comment in this thread, open feature request, or create an RFC.

Historical Roadmap:

• vLLM Ascend Roadmap Q2 2025 #448
• vLLM Ascend Roadmap Q1 2025 #71

[huyz-git]

vllm-ascend/vllm_ascend/utils.py

Lines 50 to 55 in 0060886

	# NOTE: Currently, we can only capture 1920 graphs at most,
	# due to the limitation of ACL graph. This number is bounded by
	# the number of streams, which is 2048, we save 128 streams
	# as a buffer.
	# Maximum number of graphs that can be captured by ACL Graph
	MAX_CAPTURE_SIZE = 1920

Are there any plan to enlarge the number of graphs that can be captured? 1920 is somehow too small for large models.

For example, even a 40 layers 14B model cannot have all the default 67 batch sizes be captured (40*67=2680). For a 64 layers 32B model running on two NPUs, it can only capture 1920/64/2=15 batch sizes.

In my experience, on NPU the performance gap between eager mode and ACL graph mode is large, and if the gap between each pair of batch sizes be captured is large, then the performance of the batch sizes between them will be impacted a lot. For example, when I running the Qwen3-235B model on 8 NPUs, it can only capture 1920/94/3=6 batch sizes, and the engine choose these sizes: [512,408,304,192,88,1]. In this case, the gap between 1 and 88 is large. I can get around 29 token/s for single request, but only ~20 token/s total for a batch of 2 requests (10 token/s each) and ~97 token/s for a batch of 10 requests (~9.7 token/s each). Although this is still faster than eager mode, batch of 2 requests has less total throughput than single request is a significant performance degradation.

In comparison, if I manually specific [32, 24, 16, 8, 4, 1] as the sizes, although batch size 2 and 10 is not captured, since the gap between 1 and 4 and between 8 and 16 is not too large, I can still get ~45 token/s total for a batch of 2 requests (~22 token/s each) and ~140 token/s total for a batch of 10 requests (~14 token/s each). If I manually specific [16 10 8 4 2 1] as the sizes, I can get ~49 and ~160 token/s respectively.

Mode	Graph BS	Single request	Batch size 2	Batch size 10
Eager	-	~5	~9	~44
ACL Graph	[512,408,304,192,88,1] (2 and 10 not captured, and gap 1-88 is large)	~29	~20	~97
ACL Graph	[32, 24, 16, 8, 4, 1] (2 and 10 not captured, but gap 1-4 / 8-16 is small)	~29	~44	~140
ACL Graph	[16 10 8 4 2 1] (2 and 10 captured)	~29	~49	~160

[yiz-liu]

Are there any plan to enlarge the number of graphs that can be captured? 1920 is somehow too small for large models.

For example, even a 40 layers 14B model cannot have all the default 67 batch sizes be captured (40*67=2680). For a 64 layers 32B model running on two NPUs, it can only capture 1920/64/2=15 batch sizes.

In my experience, on NPU the performance gap between eager mode and ACL graph mode is large, and if the gap between each pair of batch sizes be captured is large, then the performance of the batch sizes between them will be impacted a lot. For example, when I running the Qwen3-235B model on 8 NPUs, it can only capture 1920/94/3=6 batch sizes, and the engine choose these sizes: [512,408,304,192,88,1]. In this case, the gap between 1 and 88 is large. I can get around 29 token/s for single request, but only ~20 token/s total for a batch of 2 requests (10 token/s each) and ~97 token/s for a batch of 10 requests (~9.7 token/s each). Although this is still faster than eager mode, batch of 2 requests has less total throughput than single request is a significant performance degradation.

In comparison, if I manually specific [32, 24, 16, 8, 4, 1] as the sizes, although batch size 2 and 10 is not captured, since the gap between 1 and 4 and between 8 and 16 is not too large, I can still get ~45 token/s total for a batch of 2 requests (~22 token/s each) and ~140 token/s total for a batch of 10 requests (~14 token/s each). If I manually specific [16 10 8 4 2 1] as the sizes, I can get ~49 and ~160 token/s respectively.

Mode Graph BS Single request Batch size 2 Batch size 10
Eager - ~5 ~9 ~44
ACL Graph [512,408,304,192,88,1] (2 and 10 not captured, and gap 1-88 is large) ~29 ~20 ~97
ACL Graph [32, 24, 16, 8, 4, 1] (2 and 10 not captured, but gap 1-4 / 8-16 is small) ~29 ~44 ~140
ACL Graph [16 10 8 4 2 1] (2 and 10 captured) ~29 ~49 ~160

Yes, we have noted this and are actively working to drive improvements on this matter. If we establish any concrete timelines or plans, we will share them here.

[huangfude]

能实现EPLB吗？

[erixyao]

I do hope that vllm-ascend can officially Ascend 300I Duo soon. My customers are quite eager to deploy it!
Thank you for considering it!

[jianzs]

能实现EPLB吗？

@huangfude Static EPLB is ready #1116 . Dynamic EPLB is in development. #1391

[hikariming]

能增加对最近的新模型：minimax hunyuan、百度的支持吗～

[huyz-git]

Yes, we have noted this and are actively working to drive improvements on this matter. If we establish any concrete timelines or plans, we will share them here.

May I ask another question: What is the difference between the ACL Graph used by vllm-ascend and the graph mode used by MindIE?

In MindIE, I can set the maxBatchSize parameter to whatever I want, and it seems that the model will always run in graph mode, and I have never seen such a performance gap between different batch sizes I requested. It seems that MindIE can handle all possible batch sizes with its graph mode, and is not capped by this 1920 limits.