Add Qwen3 0.6B, 1.7B, and 4B

[jackzhxng]

Add ExecuTorch support for Qwen3 0.6B, 1.7B, and 4B

Qwen3 0.6B

Export with xnnpack + 8da4w quantization

python -m examples.models.llama.export_llama --model qwen3-0_6b --params examples/models/qwen3/0_6b_config.json -kv --use_sdpa_with_kv_cache -X --xnnpack-extended-ops -d fp32 --output_name="qwen3-0_6b_x_8da4w.pte" --verbose -qmode 8da4w

Run with pybindings

python -m examples.models.llama.runner.native --model qwen3-0_6b --pte qwen3-0_6b_x_8da4w.pte  --tokenizer ~/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/a9c98e602b9d36d2a2f7ba1eb0f5f31e4e8e5143/tokenizer.json --tokenizer_config ~/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/a9c98e602b9d36d2a2f7ba1eb0f5f31e4e8e5143/tokenizer_config.json --prompt "Who is the president of the US?" --params examples/models/qwen3/0_6b_config.json --max_len 128 -kv --temperature 0.6

>> Okay, let's see. The user is asking about the president of the US, but they wrote "And who is the president of the US?" and "And who is the president of the US?" So maybe they are using the same question but in a different way. They might be referring to the same president. Let me check. ...

# Some rough stats
Prefill time: 0.24 s
Generation tok/s: 17.15 s
Memory: 826.68 MB

Qwen3 1.7B

Export with xnnpack + 8da4w quantization

python -m examples.models.llama.export_llama --model qwen3-1_7b --params examples/models/qwen3/1_7b_config.json -kv --use_sdpa_with_kv_cache -X --xnnpack-extended-ops -d fp32 --output_name="qwen3-1_7b_x_8da4w.pte" --verbose -qmode 8da4w

Run with pybindings

python -m examples.models.llama.runner.native --model qwen3-1_7b --pte qwen3-1_7b_x_8da4w.pte  --tokenizer ~/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/a9c98e602b9d36d2a2f7ba1eb0f5f31e4e8e5143/tokenizer.json --tokenizer_config ~/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/a9c98e602b9d36d2a2f7ba1eb0f5f31e4e8e5143/tokenizer_config.json --prompt "Who is the president of the US?" --params examples/models/qwen3/1_7b_config.json --max_len 128 -kv --temperature 0.6

# Some rough stats
Prefill time: 0.25 s
Generation tok/s: 16.87 s
Memory: 1.02 GB

Qwen3 4B

Export with xnnpack + 8da4w quantization

python -m examples.models.llama.export_llama --model qwen3-4b --params examples/models/qwen3/4b_config.json -kv --use_sdpa_with_kv_cache -X --xnnpack-extended-ops -d fp32 --output_name="qwen3-4b_x_8da4w.pte" --verbose -qmode 8da4w

Run with pybindings

python -m examples.models.llama.runner.native --model qwen3-4b --pte qwen3-4b_x_8da4w.pte  --tokenizer ~/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/a9c98e602b9d36d2a2f7ba1eb0f5f31e4e8e5143/tokenizer.json --tokenizer_config ~/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/a9c98e602b9d36d2a2f7ba1eb0f5f31e4e8e5143/tokenizer_config.json --prompt "Who is the president of the US?" --params examples/models/qwen3/4b_config.json --max_len 128 -kv --temperature 0.6

# Some rough stats
Prefill time: 0.44 s
Generation tok/s: 12.12 s
Memory: 2.5 GB

bypass-github-export-checks

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10539

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 21 Pending

As of commit 626a0f0 with merge base 32dffbc ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[cbilgin]

Is there a copy paste issue or the response to the prompt somehow wrong? Looks like what's vibe coding is the prompt but the response is about US president?

[tarun292]

Please check with @madhu-fb about the qk_norm changes before landing.

[tarun292]

Is there a copy paste issue or the response to the prompt somehow wrong? Looks like what's vibe coding is the prompt but the response is about US president?

Yea same question, seems like you might not have copied over the full response?

[tarun292]

Please add a README.md for the export flow and also i don't see the config json's for 1.7B and 4B.

[mergennachin]

Nice job

Add this to the top-level README - https://github.com/pytorch/executorch/blob/main/README.md?plain=1#L54

The perf benchmarks on desktop is not really representative. As next steps, having instructions on running on iOS and Android phones, and showing the benchmarks would be good. We can publicize the actual screencast on mobile phones more.

[facebook-github-bot]

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[swolchok]

might've been cleaner to make it qk_norm_mode that can be one of None, BEFORE_ROPE, AFTER_ROPE. don't know if we have backward compatibility constraints preventing that

[tarun292]

Yea i agree, this would be cleaner. @jackzhxng let's follow up with this change.