Qwen3 Mismatch

[Helw150]

Something is mildly different between our Qwen3 implementation and the VLLM Qwen3 implementation. It's not significant enough that models transferred between the two degenerate completely but its enough to make the model significantly dumber

Plan of Attack

• Set up a full Qwen Roundtrip test
• Remove any source of numeric differences between Levanter and HF Qwen3
• Once the difference is identified, we'll need to submit a PR to VLLM to support whatever our architecture difference is so that people can inference Marin 32B with it.

[dlwh]

do we still believe this?

[Helw150]

There is still a numerical error for Qwen3 0.6B, so there is something but not the root cause for the perf we saw. Edited the issue, but leaving open until we have a passing 0.6B roundtrip!

[WhenWen]

I think I noticed one potential issue.

In

levanter/src/levanter/models/qwen.py

Line 132 in 8d1466d

use_bias=True, # Qwen2 qkv projections always have bias

We always enable attention bias

use_bias=True, # Qwen2 qkv projections always have bias

For Qwen3 this should be turned off