[DSV3] Adding deepseek-v3 model into torchtitan #1373
Conversation
facebook-github-bot
added
the
CLA Signed
|
It's possible that We actually encountered this issue when implementing TP with DeepSeek v3 models (ccing @jquesnelle who wrote most of the TP impl), and I've made a diagram to help us debug this gradient flow issue, maybe it'll help here too. |
| [float8] | ||
| enable_fsdp_float8_all_gather = false | ||
| precompute_float8_dynamic_scale_for_fsdp = false | ||
| filter_fqns = ["output"] |
There was a problem hiding this comment.
| filter_fqns = ["output"] | |
| filter_fqns = ["output", "router.gate"] |
torchtitan/models/deepseek_v3/train_configs/deepseek_v3_16b.toml
Outdated
Show resolved
Hide resolved
|
@bloc97 Thank you very much for bringing this issue to our concern! We haven't looked closely enough to the issue -- we will! However, from some initial numerical testing, it seems our TP gives the exact same numerics compared with FSDP I wonder if you have identified the same issue in our implementation, or it is more like a warning for us to be cautious in doing TP? We appreciate your feedback! |
Because how similar both implementations are, I'm assuming the same problem will show up. Training divergences only showed up in our case after training for more than 1000 steps, which logically makes sense because of how small the contribution from k_pe is. The forward pass is correct, as k_pe is computed from an unsharded tensor, and expanded into a sharded tensor (doesn't matter because all the shards are identical). However the backward pass is wrong, because it is summing a sharded tensor k (whose gradient is not the same across different shards) into an unsharded tensor k_pe. If you use TP across two gpus, GPU0's k_pe will never see the gradients of GPU1's k_pe, and vice versa! The best way to verify correctness is to check if the gradients in GPU0 are bit-identical to the gradients in GPU1 when doing TP. Pointing this out to hopefully prevent the same headache for you guys, this insidious bug was very hard to find. |
Thank you so much for pointing this out, that helps a lot! Now I see the problem with expanding k_pe from 1 head into |
|
@bloc97 We realized our code has the same issue, and we appreciate your warning a lot! I think the "root cause" is we convert DTensors to plain Tensors outside the The reason we are not using DTensors in between linear modules is because a notorious bug between complex number multiplication and PyTorch Tensor subclass. cc @bdhirsh In terms of solution, we wanted the code to be in certain style, as autograd functions would break |
@tianyu-l @wwwjn I didn't look closely where the issue pops, but assuming this statement is correct, IMO we should really fix the mentioned tensor subclass + complex number bug directly. It seems hurting us for a couple of times already so it is better to fix this in core (I think it would also benefit other tensor subclasses as a whole). cc @albanD @ezyang |
|
@wanchaol E.g. with this issue, we can't run Sequence Parallel on uneven sequence length, which means we can't do generation with Sequence Parallel, unless users explicitly handle padding / unpadding themselves. See
Moreover, this is not about adding support for something new; this is about fixing a bug between Tensor subclass and complex numbers (both are important components themselves), without which user could hit silent numerical errors. |
|
I can confirm @ezyang 's PR pytorch/pytorch#158030 fixed the DTensor + complex number bug. What a life saver! |
## Contents 1. Attention module 2. MoE module (note: I only implemented the naive routing, not the "node limit routing" strategy) 3. Deepseek-V3 model Reference: 1. Deepseek-ai: https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py 4. Huggingface: https://huggingface.co/deepseek-ai/DeepSeek-V3-Base/blob/main/modeling_deepseek.py 5. torchtitan/experiment/deepseek-v3 6. torchtitan/experiment/llama4 ## TODO - [ ] Further clean up the DeepseekV3ModelArgs class, remove unused model args - [ ] Test forward pass w/ torchtitan
Command to run: `NGPU=1 CONFIG_FILE="./torchtitan/models/deepseek_v3/train_configs/debug_model.toml" ./run_train.sh` ## Context 1. Added model args for 4 model settings, and training config for debug model 2. Debugged the forward pass, and the backward pass works out of pocket. 3. Reused c4-test dataset, and tiktokenizer from llama3 model for current testing 
…6B model (#1330) ## Context 1. Introduced a basic DSV3-16B model training config 2. Enabled FSDP/HSDP on DSV3-16B model training ## Performance Current profiler looks like this: The `to_copy` takes to long and needs to be optimized. The copy comes from dtype conversion in class MoE(): ```routed_output = (routed_output.to(torch.float32) * top_scores.unsqueeze(-1)).to(x.dtype)``` With FSDP only: <img width="1544" alt="Screenshot 2025-06-23 at 2 10 20 PM" src="https://github.com/user-attachments/assets/bcd698dc-3899-46e0-ae53-e7f8b0db13fc" />
Mostly adapted from llama4, change the TP plan based on the difference between deepseek-v3 and llama. Thanks @tianyu-l for the detailed walk through about deepseek-v3 attention model and TP plan! This diff is currently based on #1324 , and we want to extract the MoE model in DSV3 and llama4 in a shared place. Now we have: 1. FSDP 2. Activation Checkpointing 3. TP 4. CP in progress (hang due to some reason) 1. Make CP work There are minor issue with the numerical verification: With deterministic seed, the loss is not identical. I used `AdamW` optimizer. 1. FSDP degree=4 (blue line) 2. FSDP degree=4, TP degree = 2 (orange line) <img width="1368" alt="Screenshot 2025-07-01 at 5 38 50 PM" src="https://github.com/user-attachments/assets/38d96d75-6868-4482-a603-b9e10c692ed9" /> With `Adam` optimizer, the loss is **exactly the same**: <img width="1368" alt="Screenshot 2025-07-02 at 1 26 32 PM" src="https://github.com/user-attachments/assets/6b501d3c-4841-42b1-95fd-3971b16a5eeb" /> --------- Co-authored-by: Tianyu Liu <lty@fb.com>
| [model] | ||
| name = "deepseek_v3" | ||
| flavor = "debugmodel" | ||
| # test tokenizer.model, for debug purpose only |
There was a problem hiding this comment.
nit: not using test tokenizer.model anymore (now tokenizer.json and tokenizer_config.json are in the folder)
|
@bloc97 Thank you again for bringing up the TP issue, @tianyu-l and I solved this problem by changing k_pe into a DTensor, so in backward, DTensor will take care of the communication across TP ranks. For example, here's the k_pe information and it's gradient information with TP=2. During forward pass, the k_pe will change to a Replicate DTensor with shape torch.Size([8, 2048, n_heads=16, 192]). Then it sill concat with k_nope (which is a Shard(2) DTensor), and the result Hopefully this change address the issue! |
Hi @wwwjn, appreciate the nice PR. I noticed in the logs that the MFU seems a bit low compared to the 30-40% MFU that llamas typically achieve. Is this expected? |
Supported Features
To be added
Test