Skip to content

add DeepseekV3 AWQ mapping #1619

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
+26 −5
Open

add DeepseekV3 AWQ mapping #1619

Changes from 2 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
77c1d55
add deepseek_v3 awq mapping
cjackal Jul 3, 2025
0434b1f
sort `AWQ_MAPPING_REGISTRY` in alphabetical order
cjackal Jul 3, 2025
11d17ec
include q_proj in mapping
brian-dellabetta Jul 14, 2025
2b1e644
comment update
brian-dellabetta Jul 14, 2025
c6ca36c
fix OR regex mapping
brian-dellabetta Jul 14, 2025
5288bec
decrease memory when calculating w_mean
brian-dellabetta Jul 15, 2025
f30da30
Merge branch 'vllm-project:main' into main
cjackal Jul 16, 2025
2bc623f
bump `transformers>=4.52` to import `Llama4Config` from `transformers…
cjackal Jul 16, 2025
10a3cc2
Update setup.py
cjackal Jul 17, 2025
a37bba3
move `Llama4Config` importation path for compatibility with wider tra…
cjackal Jul 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions src/llmcompressor/modifiers/awq/mappings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,9 +116,25 @@ class AWQMapping:
),
]

# DeepseekV3
_deepseek_mappings = [
AWQMapping(
"re:.*input_layernorm$",
["re:.*q_a_proj$", "re:.*kv_a_proj_with_mqa$"],
),
AWQMapping("re:.*q_a_layernorm$", ["re:.*q_b_proj$"]),
AWQMapping("re:.*kv_a_layernorm$", ["re:.*kv_b_proj$"]),
AWQMapping(
"re:.*post_attention_layernorm$",
["re:.*gate_proj$", "re:.*up_proj$"],
),
AWQMapping("re:.*up_proj$", ["re:.*down_proj$"]),
]

AWQ_MAPPING_REGISTRY: Dict[str, list[AWQMapping]] = {
"CohereForCausalLM": _cohere_mappings,
"Cohere2ForCausalLM": _cohere_mappings,
"DeepseekV3ForCausalLM": _deepseek_mappings,
"Gemma2ForCausalLM": _gemma_mappings,
"Gemma3ForCausalLM": _gemma_mappings,
"Gemma3ForConditionalGeneration": _gemma_mappings,
Expand Down