Skip to content

feat: enable FP8 quantized models loading #316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

feat: enable FP8 quantized models loading #316

wants to merge 7 commits into from
+49 −26

Conversation

rafvasq
Copy link
Collaborator

@rafvasq rafvasq commented Jul 16, 2025 •
edited
Loading

Description

  • "Enable" FP8 quantized models to load via fms_mo
  • Updates example to include quantization flag
  • Adds quantization flag (unused for now) in generate_spyre_vllm_output

@rafvasq
Some updates for quant testing
29f7021 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint --active --inexact

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

@rafvasq rafvasq changed the title [DRAFT] Quantized model testing [WIP] Quantized model testing Jul 16, 2025
rafvasq added 3 commits July 22, 2025 16:14
@rafvasq
Add fp8 config loading
5efcfaa 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
@rafvasq
Lint
2df1c23 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
@rafvasq
Refactor
dfd2376 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
@rafvasq rafvasq changed the title [WIP] Quantized model testing feat: enable FP8 quantized models loading Jul 22, 2025
@rafvasq
Update tests
9762f56 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
@rafvasq rafvasq marked this pull request as ready for review July 22, 2025 21:13
@rafvasq
Lint
e541cc8 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
maxdebayser
maxdebayser reviewed Jul 23, 2025
View reviewed changes
vllm_spyre/platform.py
@@ -40,7 +40,7 @@ class SpyrePlatform(Platform):
# "spyre" device_name no longer worked due to https://github.com/vllm-project/vllm/pull/16464
device_name: str = "cpu"
device_type: str = "cpu"
supported_quantization: list[str] = ["gptq"]
supported_quantization: list[str] = ["gptq", "fp8", "compressed-tensors"]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a mention to compressed-tensors anywhere else in this PR.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe worth a comment linking to https://github.com/foundation-model-stack/fms-model-optimizer/pull/154/files#diff-1fb88c10872b0f03f1d0f6a00cd20328cdbb5e0e8bc53aa623be49b9ee3efe57R4-R9

sounds like fp8 support in fms-mo is focused on compressed-tensors

@rafvasq
Add comment
bb17bbf 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
@rafvasq rafvasq requested a review from maxdebayser July 23, 2025 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@joerunde joerunde joerunde left review comments

@yannicks1 yannicks1 Awaiting requested review from yannicks1 yannicks1 is a code owner

@prashantgupta24 prashantgupta24 Awaiting requested review from prashantgupta24 prashantgupta24 is a code owner

@sducouedic sducouedic Awaiting requested review from sducouedic sducouedic is a code owner

@tdoublep tdoublep Awaiting requested review from tdoublep tdoublep is a code owner

@nikolaospapandreou nikolaospapandreou Awaiting requested review from nikolaospapandreou nikolaospapandreou is a code owner

@maxdebayser maxdebayser Awaiting requested review from maxdebayser

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@rafvasq @maxdebayser @joerunde