Skip to content

Add all fbgemm kernel Tensors into Int4WeightOnlyConfig and Float8DynamicActivationInt4WeightConfig #2474

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: jerryzh168/stack/9
Choose a base branch
from
Open

Add all fbgemm kernel Tensors into Int4WeightOnlyConfig and Float8DynamicActivationInt4WeightConfig #2474

wants to merge 1 commit into from
+176 −120

Conversation

jerryzh168
Copy link
Contributor

@jerryzh168 jerryzh168 commented Jul 2, 2025 •
edited
Loading

Stacked PRs:

Add all fbgemm kernel Tensors into Int4WeightOnlyConfig and Float8DynamicActivationInt4WeightConfig

Summary:

  • we'd like to categorize things to derived dtype + packed format, e.g. int4 preshuffled, float8 plain
  • Added PackingFormat that has preshuffled, plain and _legacy for legacy implementation

Next:

  • deprecate FbgemmConfig since it's a single kernel (later).
  • use granularity for config API?

Test Plan:
python test/quantization/quantize_/workflows/int4/test_int4_tensor.py
python test/quantization/quantize_/workflows/int4/test_int4_preshuffled_tensor.py
python test/quantization/quantize_/workflows/float8/test_float8_tensor.py

Reviewers:

Subscribers:

Tasks:

Tags:

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jul 2, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2474

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 1 Unrelated Failure

As of commit a257b14 with merge base 378e179 (image):

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@jerryzh168 jerryzh168 force-pushed the jerryzh168/stack/10 branch from a3d0835 to 4b0c7c7 Compare July 2, 2025 01:58
jerryzh168 added a commit that referenced this pull request Jul 2, 2025
@jerryzh168
Add fbgemm preshuffle kernel into Int4WeightOnlyConfig and Float8Dyna…
4b0c7c7 
…micActivationInt4WeightConfig

Summary:
att, we will deprecate FbgemmConfig since it's a single kernel.
we'd like to categorize things to derived dtype + packed format

Test Plan:
python test/quantization/quantize_/test_int4_groupwise_preshuffle.py

Reviewers:

Subscribers:

Tasks:

Tags:

stack-info: PR: #2474, branch: jerryzh168/stack/10
This was referenced Jul 2, 2025
Merged
Merged
Merged
Open
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 2, 2025
@jerryzh168 jerryzh168 added the topic: new feature Use this tag if this PR adds a new feature label Jul 2, 2025
@jerryzh168 jerryzh168 changed the base branch from jerryzh168/stack/9 to main July 2, 2025 20:35
jerryzh168 added a commit that referenced this pull request Jul 2, 2025
@jerryzh168
Add fbgemm preshuffle kernel into Int4WeightOnlyConfig and Float8Dyna…
f5977ce 
…micActivationInt4WeightConfig

Summary:
att, we will deprecate FbgemmConfig since it's a single kernel.
we'd like to categorize things to derived dtype + packed format

Test Plan:
python test/quantization/quantize_/test_int4_groupwise_preshuffle.py

Reviewers:

Subscribers:

Tasks:

Tags:

stack-info: PR: #2474, branch: jerryzh168/stack/10
@jerryzh168 jerryzh168 force-pushed the jerryzh168/stack/10 branch from 4b0c7c7 to f5977ce Compare July 2, 2025 20:36
@jerryzh168 jerryzh168 changed the base branch from main to jerryzh168/stack/9 July 2, 2025 20:36
@jerryzh168 jerryzh168 changed the base branch from jerryzh168/stack/9 to main July 2, 2025 21:42
jerryzh168 added a commit that referenced this pull request Jul 2, 2025
@jerryzh168
Add fbgemm preshuffle kernel into Int4WeightOnlyConfig and Float8Dyna…
04ce2c5 
…micActivationInt4WeightConfig

Summary:
att, we will deprecate FbgemmConfig since it's a single kernel.
we'd like to categorize things to derived dtype + packed format

Test Plan:
python test/quantization/quantize_/test_int4_groupwise_preshuffle.py

Reviewers:

Subscribers:

Tasks:

Tags:

stack-info: PR: #2474, branch: jerryzh168/stack/10
@jerryzh168 jerryzh168 force-pushed the jerryzh168/stack/10 branch 2 times, most recently from 04ce2c5 to afd8703 Compare July 2, 2025 21:42
jerryzh168 added a commit that referenced this pull request Jul 2, 2025
@jerryzh168
Add fbgemm preshuffle kernel into Int4WeightOnlyConfig and Float8Dyna…
afd8703 
…micActivationInt4WeightConfig

Summary:
att, we will deprecate FbgemmConfig since it's a single kernel.
we'd like to categorize things to derived dtype + packed format

Test Plan:
python test/quantization/quantize_/test_int4_groupwise_preshuffle.py

Reviewers:

Subscribers:

Tasks:

Tags:

stack-info: PR: #2474, branch: jerryzh168/stack/10
@jerryzh168 jerryzh168 mentioned this pull request Jul 2, 2025
Merged
@jerryzh168 jerryzh168 changed the base branch from main to jerryzh168/stack/9 July 2, 2025 21:42
@jerryzh168 jerryzh168 changed the base branch from jerryzh168/stack/9 to main July 2, 2025 23:44
jerryzh168 added a commit that referenced this pull request Jul 2, 2025
@jerryzh168
Add fbgemm preshuffle kernel into Int4WeightOnlyConfig and Float8Dyna…
ff4682e 
…micActivationInt4WeightConfig

Summary:
att, we will deprecate FbgemmConfig since it's a single kernel.
we'd like to categorize things to derived dtype + packed format

Test Plan:
python test/quantization/quantize_/test_int4_groupwise_preshuffle.py

Reviewers:

Subscribers:

Tasks:

Tags:

stack-info: PR: #2474, branch: jerryzh168/stack/10
@jerryzh168 jerryzh168 force-pushed the jerryzh168/stack/10 branch from afd8703 to ff4682e Compare July 2, 2025 23:44
@jerryzh168 jerryzh168 changed the base branch from main to jerryzh168/stack/9 July 2, 2025 23:44
@jerryzh168 jerryzh168 changed the base branch from jerryzh168/stack/9 to main July 3, 2025 00:09
jerryzh168 added a commit that referenced this pull request Jul 3, 2025
@jerryzh168
Add fbgemm preshuffle kernel into Int4WeightOnlyConfig and Float8Dyna…
58f8a2a 
…micActivationInt4WeightConfig

Summary:
att, we will deprecate FbgemmConfig since it's a single kernel.
we'd like to categorize things to derived dtype + packed format

Test Plan:
python test/quantization/quantize_/test_int4_groupwise_preshuffle.py

Reviewers:

Subscribers:

Tasks:

Tags:

stack-info: PR: #2474, branch: jerryzh168/stack/10
@jerryzh168 jerryzh168 force-pushed the jerryzh168/stack/10 branch from ff4682e to 58f8a2a Compare July 3, 2025 00:09
@jerryzh168 jerryzh168 changed the base branch from main to jerryzh168/stack/9 July 3, 2025 00:09
@jerryzh168 jerryzh168 changed the base branch from jerryzh168/stack/9 to main July 3, 2025 02:18
jerryzh168 added a commit that referenced this pull request Jul 3, 2025
@jerryzh168
Add all fbgemm kernel Tensors into Int4WeightOnlyConfig and Float8Dyn…
532a7e1 
…amicActivationInt4WeightConfig

Summary:
att, we will deprecate FbgemmConfig since it's a single kernel.
we'd like to categorize things to derived dtype + packed format

Test Plan:
python test/quantization/quantize_/test_int4_groupwise_preshuffle.py

Reviewers:

Subscribers:

Tasks:

Tags:

stack-info: PR: #2474, branch: jerryzh168/stack/10
jerryzh168 added a commit that referenced this pull request Jul 11, 2025
@jerryzh168
Add all fbgemm kernel Tensors into Int4WeightOnlyConfig and Float8Dyn…
df9e141 
…amicActivationInt4WeightConfig

Summary:
att, we will deprecate FbgemmConfig since it's a single kernel.
we'd like to categorize things to derived dtype + packed format

Test Plan:
python test/quantization/quantize_/test_int4_groupwise_preshuffle.py

Reviewers:

Subscribers:

Tasks:

Tags:

stack-info: PR: #2474, branch: jerryzh168/stack/10
@jerryzh168 jerryzh168 force-pushed the jerryzh168/stack/10 branch from 4a3c77e to df9e141 Compare July 11, 2025 23:05
@jerryzh168 jerryzh168 changed the base branch from main to jerryzh168/stack/9 July 11, 2025 23:06
@jerryzh168 jerryzh168 changed the base branch from jerryzh168/stack/9 to main July 11, 2025 23:45
jerryzh168 added a commit that referenced this pull request Jul 11, 2025
@jerryzh168
Add all fbgemm kernel Tensors into Int4WeightOnlyConfig and Float8Dyn…
d0b8732 
…amicActivationInt4WeightConfig

Summary:
att, we will deprecate FbgemmConfig since it's a single kernel.
we'd like to categorize things to derived dtype + packed format

Test Plan:
python test/quantization/quantize_/test_int4_groupwise_preshuffle.py

Reviewers:

Subscribers:

Tasks:

Tags:

stack-info: PR: #2474, branch: jerryzh168/stack/10
@jerryzh168 jerryzh168 force-pushed the jerryzh168/stack/10 branch from df9e141 to d0b8732 Compare July 11, 2025 23:46
@jerryzh168 jerryzh168 changed the base branch from main to jerryzh168/stack/9 July 11, 2025 23:46
@jerryzh168 jerryzh168 closed this Jul 12, 2025
@jerryzh168 jerryzh168 reopened this Jul 12, 2025
@jerryzh168 jerryzh168 changed the base branch from jerryzh168/stack/9 to main July 12, 2025 03:37
jerryzh168 added a commit that referenced this pull request Jul 12, 2025
@jerryzh168
Add all fbgemm kernel Tensors into Int4WeightOnlyConfig and Float8Dyn…
cb27f24 
…amicActivationInt4WeightConfig

Summary:
att, we will deprecate FbgemmConfig since it's a single kernel.
we'd like to categorize things to derived dtype + packed format

Test Plan:
python test/quantization/quantize_/test_int4_groupwise_preshuffle.py

Reviewers:

Subscribers:

Tasks:

Tags:

stack-info: PR: #2474, branch: jerryzh168/stack/10
@jerryzh168 jerryzh168 force-pushed the jerryzh168/stack/10 branch from d0b8732 to cb27f24 Compare July 12, 2025 03:37
@jerryzh168 jerryzh168 changed the base branch from main to jerryzh168/stack/9 July 12, 2025 03:37
@jerryzh168 jerryzh168 changed the base branch from jerryzh168/stack/9 to main July 12, 2025 04:37
jerryzh168 added a commit that referenced this pull request Jul 12, 2025
@jerryzh168
Add all fbgemm kernel Tensors into Int4WeightOnlyConfig and Float8Dyn…
b82465e 
…amicActivationInt4WeightConfig

Summary:
att, we will deprecate FbgemmConfig since it's a single kernel.
we'd like to categorize things to derived dtype + packed format

Test Plan:
python test/quantization/quantize_/test_int4_groupwise_preshuffle.py

Reviewers:

Subscribers:

Tasks:

Tags:

stack-info: PR: #2474, branch: jerryzh168/stack/10
@jerryzh168 jerryzh168 force-pushed the jerryzh168/stack/10 branch from cb27f24 to b82465e Compare July 12, 2025 04:37
@jerryzh168 jerryzh168 changed the base branch from main to jerryzh168/stack/9 July 12, 2025 04:37
@jerryzh168 jerryzh168 requested a review from andrewor14 July 12, 2025 04:38
@jerryzh168 jerryzh168 changed the base branch from jerryzh168/stack/9 to main July 12, 2025 04:51
jerryzh168 added a commit that referenced this pull request Jul 12, 2025
@jerryzh168
Add all fbgemm kernel Tensors into Int4WeightOnlyConfig and Float8Dyn…
9428efc 
…amicActivationInt4WeightConfig

Summary:
we will
* deprecate FbgemmConfig since it's a single kernel (later).
* we'd like to categorize things to derived dtype + packed format, e.g. int4 preshuffled, float8 plain
* Added PackingFormat that has preshuffled, plain and _legacy for legacy implementation

Test Plan:
python test/quantization/quantize_/workflows/int4/test_int4_tensor.py
python test/quantization/quantize_/workflows/int4/test_int4_preshuffled_tensor.py
python test/quantization/quantize_/workflows/float8/test_float8_tensor.py

Reviewers:

Subscribers:

Tasks:

Tags:

stack-info: PR: #2474, branch: jerryzh168/stack/10
@jerryzh168 jerryzh168 force-pushed the jerryzh168/stack/10 branch from b82465e to 9428efc Compare July 12, 2025 04:51
@jerryzh168 jerryzh168 changed the base branch from main to jerryzh168/stack/9 July 12, 2025 04:51
@jerryzh168 jerryzh168 changed the base branch from jerryzh168/stack/9 to main July 12, 2025 04:55
jerryzh168 added a commit that referenced this pull request Jul 12, 2025
@jerryzh168
Add all fbgemm kernel Tensors into Int4WeightOnlyConfig and Float8Dyn…
dbdc2ab 
…amicActivationInt4WeightConfig

Summary:
we will
* deprecate FbgemmConfig since it's a single kernel (later).
* we'd like to categorize things to derived dtype + packed format, e.g. int4 preshuffled, float8 plain
* Added PackingFormat that has preshuffled, plain and _legacy for legacy implementation

Test Plan:
python test/quantization/quantize_/workflows/int4/test_int4_tensor.py
python test/quantization/quantize_/workflows/int4/test_int4_preshuffled_tensor.py
python test/quantization/quantize_/workflows/float8/test_float8_tensor.py

Reviewers:

Subscribers:

Tasks:

Tags:

stack-info: PR: #2474, branch: jerryzh168/stack/10
@jerryzh168 jerryzh168 force-pushed the jerryzh168/stack/10 branch from 9428efc to dbdc2ab Compare July 12, 2025 04:55
@jerryzh168 jerryzh168 changed the base branch from main to jerryzh168/stack/9 July 12, 2025 04:55
@jerryzh168
Add all fbgemm kernel Tensors into Int4WeightOnlyConfig and Float8Dyn…
a257b14 
…amicActivationInt4WeightConfig

Summary:
we will
* deprecate FbgemmConfig since it's a single kernel (later).
* we'd like to categorize things to derived dtype + packed format, e.g. int4 preshuffled, float8 plain
* Added PackingFormat that has preshuffled, plain and _legacy for legacy implementation

Test Plan:
python test/quantization/quantize_/workflows/int4/test_int4_tensor.py
python test/quantization/quantize_/workflows/int4/test_int4_preshuffled_tensor.py
python test/quantization/quantize_/workflows/float8/test_float8_tensor.py

Reviewers:

Subscribers:

Tasks:

Tags:

stack-info: PR: #2474, branch: jerryzh168/stack/10
@jerryzh168 jerryzh168 changed the base branch from jerryzh168/stack/9 to main July 12, 2025 05:03
@jerryzh168 jerryzh168 force-pushed the jerryzh168/stack/10 branch from dbdc2ab to a257b14 Compare July 12, 2025 05:03
@jerryzh168 jerryzh168 changed the base branch from main to jerryzh168/stack/9 July 12, 2025 05:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewor14 andrewor14 Awaiting requested review from andrewor14

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. topic: new feature Use this tag if this PR adds a new feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jerryzh168 @andrewor14 @facebook-github-bot