Newest Update READMe.md #1661

jonnyelisha · 2025-07-18T16:47:24Z

SUMMARY:
This PR updates the README Step 2 to reflect the new API for sparsification, fine-tuning, and quantization, replacing the old apply() function usage. Also adds explanations to improve clarity.

github-actions · 2025-07-18T16:47:39Z

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

gemini-code-assist

Summary of Changes

Hello @jonnyelisha, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request updates the README.md for the quantization_2of4_sparse_w4a16 example to reflect a significant change in the underlying API for model compression. The previous single apply() function has been replaced by a new, more granular approach that separates sparsification, fine-tuning, and quantization into distinct steps, enhancing clarity and guiding users through the updated workflow.

Highlights

API Update: The monolithic apply() function for model compression has been replaced with a more modular API, utilizing separate oneshot() and train() functions from llmcompressor.
Multi-Stage Compression Workflow: The model compression process is now explicitly broken down into three distinct and sequential stages: sparsification, fine-tuning, and quantization. Each stage is demonstrated with its own code block and saves intermediate outputs.
Improved Documentation Clarity: The README.md has been significantly updated to provide clearer explanations, including new imports, detailed parameter definitions, and a rephrased section on custom quantization strategies (tensor, group, channel).

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request updates the README to reflect a new multi-stage API for model compression, replacing the previous apply function. The changes improve clarity by breaking down the process. My review focuses on ensuring the code examples in the README are consistent and easy for users to follow, and I've provided suggestions to resolve inconsistencies between parameter definitions and their usage in the code snippets.

examples/quantization_2of4_sparse_w4a16/README.md

gemini-code-assist · 2025-07-18T16:49:37Z

examples/quantization_2of4_sparse_w4a16/README.md

+oneshot(
    model=model,
    dataset=dataset,
    recipe=recipe,
-    bf16=False,  # use full precision for training
+    splits=splits,
+    num_calibration_samples=512,
+    preprocessing_num_workers=8,
    output_dir=output_dir,
+    stage="sparsity_stage",
+)
+
+# 2. Sparse fine-tuning: improve accuracy on pruned model
+train(
+    model=output_path / "sparsity_stage",
+    dataset=dataset,
+    recipe=recipe,
    splits=splits,
-    max_seq_length=512,
    num_calibration_samples=512,
+    preprocessing_num_workers=8,
+    bf16=False,
+    max_seq_length=512,
    num_train_epochs=0.5,
    logging_steps=500,
    save_steps=5000,
    gradient_checkpointing=True,
    learning_rate=0.0001,
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
+    output_dir=output_dir,
+    stage="finetuning_stage",
+)
+
+# 3. Oneshot quantization: compress model weights to lower precision
+quantized_model = oneshot(
+    model=output_path / "finetuning_stage",
+    dataset=dataset,
+    recipe=recipe,
+    splits=splits,
+    num_calibration_samples=512,
+    preprocessing_num_workers=8,
+    output_dir=output_dir,
+    stage="quantization_stage",
+)


This code block uses hardcoded values for many parameters that are already defined as variables in the Step 1 code block. Using the variables will make the example cleaner, easier to understand, and less prone to inconsistencies if the parameters in Step 1 are changed. This also makes it easier for users to copy, paste, and modify the example.

Note: This suggestion is dependent on applying the suggested changes to the parameter definitions in Step 1.

oneshot( model=model, dataset=dataset, recipe=recipe, splits=splits, num_calibration_samples=num_calibration_samples, preprocessing_num_workers=preprocessing_num_workers, output_dir=output_dir, stage="sparsity_stage", ) # 2. Sparse fine-tuning: improve accuracy on pruned model train( model=output_path / "sparsity_stage", dataset=dataset, recipe=recipe, splits=splits, num_calibration_samples=num_calibration_samples, preprocessing_num_workers=preprocessing_num_workers, bf16=bf16, max_seq_length=max_seq_length, num_train_epochs=num_train_epochs, logging_steps=logging_steps, save_steps=save_steps, gradient_checkpointing=gradient_checkpointing, learning_rate=learning_rate, lr_scheduler_type=lr_scheduler_type, warmup_ratio=warmup_ratio, output_dir=output_dir, stage="finetuning_stage", ) # 3. Oneshot quantization: compress model weights to lower precision quantized_model = oneshot( model=output_path / "finetuning_stage", dataset=dataset, recipe=recipe, splits=splits, num_calibration_samples=num_calibration_samples, preprocessing_num_workers=preprocessing_num_workers, output_dir=output_dir, stage="quantization_stage", )

Signed-off-by: Jonathan <jonathanelisha98@gmail.com>

dsikka

Thank you! One comment

dsikka · 2025-07-21T13:13:43Z

examples/quantization_2of4_sparse_w4a16/README.md

+    output_dir=output_dir,
+    stage="quantization_stage",
+)
+quantized_model.save_pretrained(


Do you mind adding one comment here about skip_sparsity_comorpession_stats being set to False here? This is done so that we make sure we're accounting for sparsity in the model when compressing it and saving it to disk.

brian-dellabetta

Thanks for the contribution @jonnyelisha ! One nit comment on formatting, but this looks great

examples/quantization_2of4_sparse_w4a16/README.md

Co-authored-by: Brian Dellabetta <brian-dellabetta@users.noreply.github.com>

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

gemini-code-assist bot reviewed Jul 18, 2025

View reviewed changes

Update README for quantization example vllm-project#1659

718afcb

Signed-off-by: Jonathan <jonathanelisha98@gmail.com>

jonnyelisha force-pushed the main branch from ebdf1fd to 718afcb Compare July 18, 2025 16:51

Merge branch 'main' into main

2f8588f

dsikka added the ready When a PR is ready for review label Jul 19, 2025

dsikka reviewed Jul 21, 2025

View reviewed changes

brian-dellabetta reviewed Jul 21, 2025

View reviewed changes

examples/quantization_2of4_sparse_w4a16/README.md Outdated Show resolved Hide resolved

jonnyelisha and others added 2 commits July 21, 2025 19:59

Update README.md

1ee8f2e

Co-authored-by: Brian Dellabetta <brian-dellabetta@users.noreply.github.com>

Update examples/quantization_2of4_sparse_w4a16/README.md

35f0da6

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Newest Update READMe.md #1661

Newest Update READMe.md #1661

jonnyelisha commented Jul 18, 2025

Uh oh!

github-actions bot commented Jul 18, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

gemini-code-assist bot Jul 18, 2025

Uh oh!

dsikka left a comment

Uh oh!

dsikka Jul 21, 2025

Uh oh!

jonnyelisha Jul 21, 2025

Uh oh!

brian-dellabetta left a comment

Uh oh!

Uh oh!

Uh oh!

Newest Update READMe.md #1661

Are you sure you want to change the base?

Newest Update READMe.md #1661

Conversation

jonnyelisha commented Jul 18, 2025

Uh oh!

github-actions bot commented Jul 18, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

gemini-code-assist bot Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

dsikka left a comment

Choose a reason for hiding this comment

Uh oh!

dsikka Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

jonnyelisha Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

brian-dellabetta left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!