Skip to content

Commit db892e7

Browse files
authored
Fix input layer norm mismatch for Eagle Speculative Decoding compatib… (#548)
* Fix input layer norm mismatch for Eagle Speculative Decoding compatibility The LLaMA decoder layer applies input layer normalization at every layer, whereas Eagle omits it for the initial layer, using a dummy InputLayerNorm class instead. Recently, LLaMA's input layer norm implementation (https://github.com/ROCm/vllm/blob/262ed1e16c5bd71f0612b700186854b8c932565d/vllm/model_executor/models/llama.py#L326) was updated to accept at most 3 inputs. To maintain compatibility and prevent Eagle Speculative Decoding from failing, this dummy class needs to be updated accordingly. * Update eagle.py
1 parent d1d3ff9 commit db892e7

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

vllm/model_executor/models/eagle.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ def __init__(self, weight=None, bias=None):
2929
self.weight = nn.Parameter(weight) if weight is not None else None
3030
self.bias = nn.Parameter(bias) if bias is not None else None
3131

32-
def forward(self, x):
32+
def forward(self, x, residual=None, scale=None):
3333
return x
3434

3535

0 commit comments

Comments
 (0)