[V1][Spec-decode] First Stage support of Eagle 1

[umeiko]

What this PR does / why we need it?

• Implements preliminary version of Eagle speculative decoding. [Feature]: Implement Eagle1 Acceleration on vllm-ascend #1088
• Lays foundation for future performance optimizations in token generation

Does this PR introduce any user-facing change?

• No user-facing changes yet (experimental implementation)
• Will add configuration options as same as vllm does.

How was this patch tested?

• Basic functionality verification with synthetic data
• Planning more comprehensive benchmarks in subsequent work

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[JSYRD]

Hello, also found eagle is still not working with v0.9.0rc2 ( looks correctly loaded but will crash), however there's a patch in /vllm_ascend/patch/worker/patch_common/patch_eagle.py, and seems that this patch might be used to support eagle in vllm-ascend(and obviously it's not working well). The problem is, is it a ideal way to implement eagle in vllm-ascend in the way like your pr(implementing a new worker), or like the patch? Not very professional in vllm-ascend developing, and still wondering the differences between those files in the worker and the patch/worker 😂

[umeiko]

• LLM: Meta-Llama-3.1-8B-Instruct
• Drafter: EAGLE-LLaMA3.1-Instruct-8B
• Dataset: mt-bench (downloaded from the eagle repo )
• num_spec_tokens 2
• vllm : matser branch
• Device: Single 910B3 , 65536 MB

========test_matrix==========
total_counts: 8994
position_0: 8994 times (1.00)
position_1: 5680 times (0.63)
position_2: 2854 times (0.32)
mean acceptance length: 1.2661774516344229

[umeiko]

Hello, also found eagle is still not working with v0.9.0rc2 ( looks correctly loaded but will crash), however there's a patch in /vllm_ascend/patch/worker/patch_common/patch_eagle.py, and seems that this patch might be used to support eagle in vllm-ascend(and obviously it's not working well). The problem is, is it a ideal way to implement eagle in vllm-ascend in the way like your pr(implementing a new worker), or like the patch? Not very professional in vllm-ascend developing, and still wondering the differences between those files in the worker and the patch/worker 😂

I updated this commit after eagle3 pr is ready. This brach has been tested on v0.9.1 locally.

[JSYRD]

Hello, also found eagle is still not working with v0.9.0rc2 ( looks correctly loaded but will crash), however there's a patch in /vllm_ascend/patch/worker/patch_common/patch_eagle.py, and seems that this patch might be used to support eagle in vllm-ascend(and obviously it's not working well). The problem is, is it a ideal way to implement eagle in vllm-ascend in the way like your pr(implementing a new worker), or like the patch? Not very professional in vllm-ascend developing, and still wondering the differences between those files in the worker and the patch/worker 😂

I updated this commit after eagle3 pr is ready. This brach has been tested on v0.9.1 locally.

Got that. But as the same question mentioned above, it seems there's already an implementation of eagle1, and is it waited to be fixed? or your version just implemented another in a different way? Thanks for your reply~

[umeiko]

Hello, also found eagle is still not working with v0.9.0rc2 ( looks correctly loaded but will crash), however there's a patch in /vllm_ascend/patch/worker/patch_common/patch_eagle.py, and seems that this patch might be used to support eagle in vllm-ascend(and obviously it's not working well). The problem is, is it a ideal way to implement eagle in vllm-ascend in the way like your pr(implementing a new worker), or like the patch? Not very professional in vllm-ascend developing, and still wondering the differences between those files in the worker and the patch/worker 😂

I updated this commit after eagle3 pr is ready. This brach has been tested on v0.9.1 locally.

Got that. But as the same question mentioned above, it seems there's already an implementation of eagle1, and is it waited to be fixed? or your version just implemented another in a different way? Thanks for your reply~

yes. eagle 1 support have not been finished in v1 kernel before. this pr fixes this.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 49.92%. Comparing base (c30ddb8) to head (1296606).
Report is 75 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1128       +/-   ##
===========================================
+ Coverage   27.39%   49.92%   +22.53%     
===========================================
  Files          56       76       +20     
  Lines        6191     9231     +3040     
===========================================
+ Hits         1696     4609     +2913     
- Misses       4495     4622      +127     

Flag	Coverage Δ
unittests	49.92% <ø> (+22.53%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[umeiko]

Serving Scripts:

MODEL=/mnt/models/Meta-Llama-3.1-8B-Instruct
EAGEL1DRAFT=/mnt/models/EAGLE-LLaMA3.1-Instruct-8B

DRAFT=$EAGEL1DRAFT

VLLM_USE_V1=1 python -m vllm.entrypoints.openai.api_server\
    --host 0.0.0.0 --port 8001 --model $MODEL --enforce_eager \
    --seed 42 -tp 1 --gpu_memory_utilization 0.8 --max_num_seqs 1 --max_model_len 1024\
    --speculative_config '{ "method": "eagle", "model": "'$DRAFT'", "draft_tensor_parallel_size": 1, "num_speculative_tokens": 2, "max_model_len": 1024 }'

Serving Benchmark On Meta-Llama-3.1-8B-Instruct

Dataset ShareGPT_V4.3_unfiltered_cleaned_split.json , 46 prompts.

• without SpecDecode

============ Serving Benchmark Result ============
Successful requests:                     46        
Benchmark duration (s):                  297.60    
Total input tokens:                      8177      
Total generated tokens:                  8608      
Request throughput (req/s):              0.15      
Output token throughput (tok/s):         28.92     
Total Token throughput (tok/s):          56.40     
---------------Time to First Token----------------
Mean TTFT (ms):                          48.65     
Median TTFT (ms):                        45.49     
P99 TTFT (ms):                           74.90     
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          34.37     
Median TPOT (ms):                        34.40     
P99 TPOT (ms):                           35.27     
---------------Inter-token Latency----------------
Mean ITL (ms):                           34.49     
Median ITL (ms):                         34.38     
P99 ITL (ms):                            36.78     
==================================================

• with eagle1

============ Serving Benchmark Result ============
Successful requests:                     46        
Benchmark duration (s):                  222.79    
Total input tokens:                      8177      
Total generated tokens:                  8608      
Request throughput (req/s):              0.21      
Output token throughput (tok/s):         38.64     
Total Token throughput (tok/s):          75.34     
---------------Time to First Token----------------
Mean TTFT (ms):                          56.12     
Median TTFT (ms):                        52.86     
P99 TTFT (ms):                           83.48     
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          27.46     
Median TPOT (ms):                        25.95     
P99 TPOT (ms):                           42.74     
---------------Inter-token Latency----------------
Mean ITL (ms):                           47.15     
Median ITL (ms):                         47.07     
P99 ITL (ms):                            49.69     
==================================================

[mengwei805]

Issues that must be resolved: Please delete the eagle code that @ponix-j did not complete in #874

[umeiko]

tests for eagle1 was opened here

[umeiko]

Issues that must be resolved: Please delete the eagle code that @ponix-j did not complete in #874

we made a single pr to fix this #1385

[mengwei805]

Issues that must be resolved: Please delete the eagle code that @ponix-j did not complete in #874

we made a single pr to fix this #1385

approve

[umeiko]

longterm CI currently not working : #1444

[umeiko]

deepseek_mtp CI need to be skip.

[MengqingCao]

Great! you can just remove the comment to enable it

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.