RuntimeError: 'weight' must be 2-D

[Hasan-Syed25]

RuntimeError: 'weight' must be 2-D occurs when I am using Deepspeed Zero3 for distributed training. Is this an issue with deepspeed or is it an initialization issue. Here is link to the same issue that I am facing. What am I missing here?

Thanks

[Crystalcareai]

Deepspeed zero3 currently throws a lot of errors - we're working on it and will have a fix out soon.

[fernando-neto-ai]

@Hasan-Syed25 could you paste your env and your code, so I can deep dive on it?

[zhangchushu]

Same problem happens to me. Has anyone solved it?

[shing100]

Same problem happens to me. Has anyone solved this problem?

[raghavgarg97]

I would like to add a +1 too

I guess its probably because deepspeed is wrapping the student model but not the teacher model

A quick fix could be to store the logits offline.

[mertege]

Deepspeed zero3 currently throws a lot of errors - we're working on it and will have a fix out soon.

FSDP also brings the same issue.

[rookiebird]

I meet the same problem and solve it by calling deepspeed.initialize for teacher_model when initialize the trainer.

[romit-sarvam]

@rookiebird Did you try with FSDP? What could be the potential solve for FSDP?

[adumans]

deepspeed stage2 works