[Model] vllm v1 support mlp_speculator #20655

skylee-01 · 2025-07-09T01:45:24Z

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

vllm v1 support mlp_speculator

Test Plan

Test Result

(Optional) Documentation Update

Signed-off-by: skylee-01 <497627264@qq.com>

gemini-code-assist

Summary of Changes

Hello @skylee-01, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces comprehensive support for mlp_speculator as a new speculative decoding method in vLLM v1. It involves updating configuration utilities to recognize the new method, adapting the MLPSpeculator model for its role as a draft model, and integrating a dedicated proposer class into the GPU model runner to manage the generation of speculative tokens.

Highlights

New Speculative Decoding Method: Added support for "mlp_speculator" as a new speculative decoding method within vLLM v1, allowing its configuration and use.
MLPSpeculator Model Adaptation: Modified the MLPSpeculator model's initialization to correctly handle configurations when used as a draft model in speculative decoding, and adjusted its generate_proposals method for conditional sampling_metadata usage.
New Proposer Class: Introduced MLPSpeculatorProposer (vllm/v1/spec_decode/mlp_speculator.py) to encapsulate the logic for generating draft tokens specifically for the MLPSpeculator model.
Integration into GPU Model Runner: Integrated MLPSpeculatorProposer into GPUModelRunner to enable the mlp_speculator method, including the necessary data preparation and call flow for proposing draft tokens.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request adds support for mlp_speculator in vLLM v1. The changes are in the right direction, but I've found several critical issues related to the integration with the v1 engine, particularly in vllm/model_executor/models/mlp_speculator.py and vllm/v1/worker/gpu_model_runner.py. These issues could lead to runtime errors or incorrect behavior. I've provided detailed suggestions to fix them. Addressing these points will be important for the stability of this new feature.

gemini-code-assist · 2025-07-09T01:48:12Z

vllm/model_executor/models/mlp_speculator.py

+        if hasattr(vllm_config, 'speculative_config'):
+            config = vllm_config.speculative_config.draft_model_config.hf_config
+            self.sampling_metadata_is_required = False
+        else:
+            config = vllm_config.model_config.hf_config
+            self.sampling_metadata_is_required = True


The logic to differentiate between v0 and v1 paths is incomplete. For the v1 path (hasattr(vllm_config, 'speculative_config')), self.sampler and self.logits_processor are still initialized with v0 components at the end of __init__ (lines 147-149), which will cause a crash due to type mismatches with SamplingMetadata.

To fix this, the initialization of sampler and logits_processor should be moved inside this if/else block. For the v1 path, self.sampler should be an instance of vllm.v1.sample.sampler.Sampler and self.logits_processor should be None.

vllm/model_executor/models/mlp_speculator.py

vllm/v1/spec_decode/mlp_speculator.py

vllm/v1/worker/gpu_model_runner.py

vllm/engine/arg_utils.py

vllm/v1/spec_decode/mlp_speculator.py

github-actions · 2025-07-09T01:49:37Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: skylee-01 <497627264@qq.com>

skylee-01 · 2025-07-09T11:07:59Z

How should pre-commit be handled?

skylee-01 · 2025-07-10T02:28:57Z

Hi Curys, please help me review this PR. Thank you very much. @DarkLight1337

DarkLight1337 · 2025-07-10T05:30:43Z

@WoosukKwon can you review this? I'm not that familiar with spec decode code

skylee-01 · 2025-07-11T01:41:35Z

Teacher, could you please help me take a look at this pull request? @WoosukKwon

skylee-01 · 2025-07-15T06:33:11Z

Teacher Lily, please help me review it. @LiuXiaoxuanPKU

[Model] vllm v1 support mlp_speculator

859e2ca

Signed-off-by: skylee-01 <497627264@qq.com>

skylee-01 requested review from WoosukKwon, robertgshaw2-redhat, njhill, ywang96, comaniac and alexm-redhat as code owners July 9, 2025 01:45

gemini-code-assist bot reviewed Jul 9, 2025

View reviewed changes

mergify bot added speculative-decoding v1 labels Jul 9, 2025

gemini-code-assist bot reviewed Jul 9, 2025

View reviewed changes

skylee-01 added 5 commits July 9, 2025 10:45

Optimize code format

2a57e20

Signed-off-by: skylee-01 <497627264@qq.com>

Optimize code format

4711051

Signed-off-by: skylee-01 <497627264@qq.com>

fix bug

26f2b7f

Signed-off-by: skylee-01 <497627264@qq.com>

Optimize code format

e7ad830

Signed-off-by: skylee-01 <497627264@qq.com>

Optimize code format

de28b86

Signed-off-by: skylee-01 <497627264@qq.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Model] vllm v1 support mlp_speculator #20655

[Model] vllm v1 support mlp_speculator #20655

skylee-01 commented Jul 9, 2025 •

edited by github-actions bot

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Jul 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

skylee-01 commented Jul 9, 2025

Uh oh!

skylee-01 commented Jul 10, 2025

Uh oh!

DarkLight1337 commented Jul 10, 2025

Uh oh!

skylee-01 commented Jul 11, 2025

Uh oh!

skylee-01 commented Jul 15, 2025

Uh oh!

Uh oh!

Uh oh!

[Model] vllm v1 support mlp_speculator #20655

Are you sure you want to change the base?

[Model] vllm v1 support mlp_speculator #20655

Conversation

skylee-01 commented Jul 9, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Essential Elements of an Effective PR Description Checklist

Purpose

Test Plan

Test Result

(Optional) Documentation Update

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

skylee-01 commented Jul 9, 2025

Uh oh!

skylee-01 commented Jul 10, 2025

Uh oh!

DarkLight1337 commented Jul 10, 2025

Uh oh!

skylee-01 commented Jul 11, 2025

Uh oh!

skylee-01 commented Jul 15, 2025

Uh oh!

Uh oh!

skylee-01 commented Jul 9, 2025 •

edited by github-actions bot

Loading