[BUG]try to train lora but the lora parameter seem not be updated, how to check the reason

[523997931]

Describe the bug
A clear and concise description of what the bug is.

here is my train code to check, I check the parameter value, and find it not change during my train, it is always 0.0, I don't know how to debug for the problem

[zhangjingxian1998]

i meet same issue, and find this. but i don't know whether right or not
change the return self.overflow position

DeepSpeed/deepspeed/runtime/fp16/fused_optimizer.py

Lines 255 to 289 in 69e03e5

	def step(self, closure=None):
	"""
	Not supporting closure.
	"""
	if self.fused_adam_legacy:
	return self.step_fused_adam()
	# First determine if there is overflow.
	if self.timers:
	self.timers(OVERFLOW_CHECK_TIMER).start()
	fp16_params = []
	for i, group in enumerate(self.fp16_groups):
	fp16_params.extend([p for p in group if p.grad is not None])
	self.overflow = self.overflow_checker.has_overflow(fp16_params)
	if self.timers:
	self.timers(OVERFLOW_CHECK_TIMER).stop()
	if self.use_grad_scaling:
	prev_scale = self.cur_scale
	self._update_scale(self.overflow)
	if self.overflow:
	if self.verbose:
	log_dist(
	"Overflow detected. Skipping step. Attempted loss "
	f"scale: {prev_scale}, reducing to {self.cur_scale}",
	ranks=[0])
	# Clear gradients
	for i, group in enumerate(self.fp16_groups):
	for p in group:
	p.grad = None
	if self.timers:
	self.timers.log(OVERFLOW_TIMERS)
	return self.overflow

to

     def step(self, closure=None): 
         """ 
         Not supporting closure. 
         """ 
  
         if self.fused_adam_legacy: 
             return self.step_fused_adam() 
  
         # First determine if there is overflow. 
         if self.timers: 
             self.timers(OVERFLOW_CHECK_TIMER).start() 
         fp16_params = [] 
         for i, group in enumerate(self.fp16_groups): 
             fp16_params.extend([p for p in group if p.grad is not None]) 
         self.overflow = self.overflow_checker.has_overflow(fp16_params) 
         if self.timers: 
             self.timers(OVERFLOW_CHECK_TIMER).stop() 
  
         if self.use_grad_scaling: 
             prev_scale = self.cur_scale 
             self._update_scale(self.overflow) 
             if self.overflow: 
                 if self.verbose: 
                     log_dist( 
                         "Overflow detected. Skipping step. Attempted loss " 
                         f"scale: {prev_scale}, reducing to {self.cur_scale}", 
                         ranks=[0]) 
                 # Clear gradients 
                 for i, group in enumerate(self.fp16_groups): 
                     for p in group: 
                         p.grad = None 
                 return self.overflow 
  
             if self.timers: 
                 self.timers.log(OVERFLOW_TIMERS) 
             

[523997931]

I use the register_full_backward_hook to print the module name and grad input and output, and I print the result like
[Backward Hook]
grad_input: [None]
grad_output: [torch.Size([1, 16, 21, 64, 100])]
[Backward Hook] 41
grad_input: [None]
grad_output: [torch.Size([1, 16, 21, 64, 100])]
[Backward Hook] 41.head.head
grad_input: [torch.Size([1, 33600, 5120])]
grad_output: [torch.Size([1, 33600, 64])]
[Backward Hook] 41.head.norm
grad_input: [torch.Size([1, 33600, 5120])]
grad_output: [torch.Size([1, 33600, 5120])]
[Backward Hook] 41.head
grad_input: [torch.Size([1, 33600, 5120]), torch.Size([1, 1, 5120])]
grad_output: [torch.Size([1, 33600, 64])]
before
[Block] input x id: 22410617771712
after
[Block] output x id: 22410617207392
True
True
[Backward Hook] 40
grad_input: [None]
grad_output: [None, torch.Size([1, 1, 5120]), torch.Size([1, 1, 6, 5120]), None, None, torch.Size([1024, 64]), torch.Size([1, 769, 5120]), torch.Size([1, 21, 32, 768]), None, None]
[Backward Hook] 40.block.ffn.2.lora_B.default
grad_input: [torch.Size([1, 33600, 16])]
grad_output: [None]
[Backward Hook] 40.block.ffn.2.lora_A.default
grad_input: [torch.Size([1, 33600, 13824])]
grad_output: [torch.Size([1, 33600, 16])]
[Backward Hook] 40.block.ffn.2.lora_dropout.default
grad_input: [torch.Size([1, 33600, 13824])]
grad_output: [torch.Size([1, 33600, 13824])]
[Backward Hook] 40.block.ffn.2.base_layer
grad_input: [torch.Size([1, 33600, 13824])]
grad_output: [None]
[Backward Hook] 40.block.ffn.2
grad_input: [torch.Size([1, 33600, 13824])]
grad_output: [None]
[Backward Hook] 40.block.ffn.1

than I find the grad become None when it to the 40 layer, [Backward Hook] 40.block.ffn.2.base_layer
grad_input: [torch.Size([1, 33600, 13824])]
grad_output: [None] but I don't know why, same the grad is not be catched