Spec decode support for V1 Engine #874

ponix-j · 2025-05-15T11:44:42Z

What this PR does / why we need it?

Does this PR introduce any user-facing change?

How was this patch tested?

mengwei805 · 2025-05-15T12:56:02Z

vllm_ascend/sample/rejection_sampler.py

@@ -0,0 +1,587 @@
+# SPDX-License-Identifier: Apache-2.0
+from typing import Optional


rejection_sampler and eagle_proposer fully overwrite,
Can't this be implemented in a way that the main body uses vllm code and the problematic part is patched in vllm_ascend?

Use class AscendRejectionSampler Inheritance class RejectionSampler，Extracting the part of the change

mengwei805 · 2025-05-15T12:59:50Z

spec decode is key feature, can u add ngram and eagle e2e ut in vllm-ascend?

ponix-j · 2025-05-16T02:53:44Z

spec decode is key feature, can u add ngram and eagle e2e ut in vllm-ascend?

yes，already added test_spec_decode.py

mengwei805 · 2025-05-17T01:58:09Z

pls rebase u all commits to 1 commit

MengqingCao · 2025-05-20T03:17:50Z

tests/singlecard/spec_decode/e2e/test_spec_decode.py

+
+@pytest.fixture
+def model_name():
+    return "meta-llama/Meta-Llama-3-8B-Instruct"


Let's use modelscope instead

Suggested change

return "meta-llama/Meta-Llama-3-8B-Instruct"

return "LLM-Research/Meta-Llama-3.1-8B-Instruct"

MengqingCao · 2025-05-20T03:18:00Z

tests/singlecard/spec_decode/e2e/test_spec_decode.py

+
+@pytest.fixture
+def eagle_model_name():
+    return "yuhuili/EAGLE-LLaMA3-Instruct-8B"


MengqingCao · 2025-05-20T03:23:16Z

vllm_ascend/patch/spec_decode/patch_common/eagle.py

@@ -0,0 +1,70 @@
+# SPDX-License-Identifier: Apache-2.0


Not recommand to make a seperate patch dir for spec-decode. Let's move these patch to vllm_ascend/patch/platform or vllm_ascend/patch/worker, depending on which period is appropriate.

And the most important, please make comments in vllm_ascend/patch/__init__.py to describe why we make this patch

allready fixed

MengqingCao · 2025-05-20T03:23:48Z

vllm_ascend/patch/spec_decode/patch_common/eagle.py

+    #                 a, a + 1, ..., a + b - n2 - 1,
+    #                 a + b, a + b + 1, ..., a + b + c - n3 - 1]
+
+    # [0, a, a + b, a + b + c] -> [a, b, c]


plz remove the useless comments

this is copy from vllm/eagle.py, why this useless?

I mistakenly thought it was a cuda-specific comment, please ignore it

MengqingCao · 2025-05-20T03:28:13Z

vllm_ascend/utils.py

@@ -100,6 +100,7 @@ def adapt_patch(is_global_patch: bool = False):
    if is_global_patch:
        from vllm_ascend.patch import platform  # noqa: F401
    else:
+        from vllm_ascend.patch import spec_decode  # noqa: F401


let's remove this when the above suggestions on patch is solved

already fixed

MengqingCao · 2025-05-20T03:36:48Z

vllm_ascend/worker/model_runner_v1.py

+        # [0, 1, 2, 5, 6, 9]
+        target_logits_indices += arange
+
+        # TODO: Optimize the CPU -> GPU copy.


Suggested change

# TODO: Optimize the CPU -> GPU copy.

# TODO: Optimize the CPU -> NPU copy.

already fixed

MengqingCao · 2025-05-20T03:39:08Z

vllm_ascend/worker/model_runner_v1.py

@@ -737,12 +888,92 @@ def execute_model(
        if max_gen_len == 1:
            # No spec decode tokens.
            valid_sampled_token_ids = sampled_token_ids.tolist()
+        else:
+            # Includes spec decode tokens.


could we extract the model execution code into a seperate function to make code clear?

already fixed

Signed-off-by: ponix-j <657511300@qq.com>

tests/singlecard/spec_decode/e2e/test_spec_decode.py

Co-authored-by: Mengqing Cao <cmq0113@163.com>

ponix-j · 2025-05-21T07:57:44Z

pls rebase u all commits to 1 commit

will use 1 commit finally

github-actions bot added the module:tests label May 15, 2025

ponix-j changed the title ~~Spec v0.8.5rc1~~ Spec decode v0.8.5rc1 May 15, 2025

mengwei805 reviewed May 15, 2025

View reviewed changes

github-actions bot added the module:core label May 16, 2025

XWFAlone mentioned this pull request May 17, 2025

[1/N][UT][v1 MTP] add basic v1 mtp features #890

Open

ponix-j force-pushed the spec_v0.8.5rc1 branch 2 times, most recently from 9f5ae8a to ed32fcf Compare May 19, 2025 04:04

github-actions bot added documentation Improvements or additions to documentation module:ops module:quantization labels May 19, 2025

ponix-j force-pushed the spec_v0.8.5rc1 branch from ed32fcf to d8848a2 Compare May 19, 2025 06:51

github-actions bot removed documentation Improvements or additions to documentation module:ops module:quantization labels May 19, 2025

ponix-j force-pushed the spec_v0.8.5rc1 branch 2 times, most recently from 9e7bc8b to aa2bb74 Compare May 20, 2025 01:16

MengqingCao reviewed May 20, 2025

View reviewed changes

github-actions bot removed the module:core label May 20, 2025

[V1] Support ngram spec decode

6d7187b

Signed-off-by: ponix-j <657511300@qq.com>

ponix-j force-pushed the spec_v0.8.5rc1 branch from 526a7c4 to 6d7187b Compare May 21, 2025 01:06

MengqingCao reviewed May 21, 2025

View reviewed changes

tests/singlecard/spec_decode/e2e/test_spec_decode.py Outdated Show resolved Hide resolved

ponix-j and others added 4 commits May 21, 2025 11:16

Update tests/singlecard/spec_decode/e2e/test_spec_decode.py

80f47df

Co-authored-by: Mengqing Cao <cmq0113@163.com>

Update vllm_ascend_test.yaml

782a5e9

Update rejection_sampler.py

71cd3cd

Update model_runner_v1.py

8cbd724

Update vllm_ascend_test.yaml

08ad20c

ponix-j added 4 commits May 21, 2025 16:47

Update test_spec_decode.py

fa2481d

Update rejection_sampler.py

0f895ed

Update test_spec_decode.py

8ccfb97

Update vllm_ascend_test.yaml

d9e18d5

wangxiyuan changed the title ~~Spec decode v0.8.5rc1~~ Spec decode support for V1 Engine May 21, 2025

ponix-j added 10 commits May 21, 2025 17:43

Rename test_spec_decode.py to test_v1_spec_decode.py

6c85953

Update vllm_ascend_test.yaml

b19c548

Update vllm_ascend_test.yaml

97436a4

Update test_v1_spec_decode.py

cbcbead

Update vllm_ascend_test.yaml

fc90dce

Update vllm_ascend_test.yaml

474ab4c

Update test_v1_spec_decode.py

da63764

Update test_v1_spec_decode.py

126269f

Update attention_v1.py

7f9ae0a

Update attention_v1.py

36c2b03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spec decode support for V1 Engine #874

Spec decode support for V1 Engine #874

ponix-j commented May 15, 2025

mengwei805 May 15, 2025

ponix-j May 21, 2025

mengwei805 commented May 15, 2025

ponix-j commented May 16, 2025

mengwei805 commented May 17, 2025

MengqingCao May 20, 2025

MengqingCao May 20, 2025

MengqingCao May 20, 2025

ponix-j May 21, 2025

MengqingCao May 20, 2025

ponix-j May 20, 2025

MengqingCao May 21, 2025

MengqingCao May 20, 2025

ponix-j May 21, 2025

MengqingCao May 20, 2025

ponix-j May 21, 2025

MengqingCao May 20, 2025

ponix-j May 21, 2025

ponix-j commented May 21, 2025

		@@ -0,0 +1,587 @@
		# SPDX-License-Identifier: Apache-2.0
		from typing import Optional

	return "meta-llama/Meta-Llama-3-8B-Instruct"
	return "LLM-Research/Meta-Llama-3.1-8B-Instruct"

	# TODO: Optimize the CPU -> GPU copy.
	# TODO: Optimize the CPU -> NPU copy.

Spec decode support for V1 Engine #874

Are you sure you want to change the base?

Spec decode support for V1 Engine #874

Conversation

ponix-j commented May 15, 2025

What this PR does / why we need it?

Does this PR introduce any user-facing change?

How was this patch tested?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mengwei805 commented May 15, 2025

ponix-j commented May 16, 2025

mengwei805 commented May 17, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ponix-j commented May 21, 2025