Skip to content

Commit 77ff27b

Browse files
authored
[bugfix] fix ascend_forward_context with model_runner_v0 (#1598)
### What this PR does / why we need it? fix bug `'AscendMetadata' object has no attribute 'num_actual_tokens'`. ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Signed-off-by: zzzzwwjj <1183291235@qq.com>
1 parent 65909b2 commit 77ff27b

File tree

1 file changed

+6
-1
lines changed

1 file changed

+6
-1
lines changed

vllm_ascend/ascend_forward_context.py

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,8 +59,13 @@ def set_ascend_forward_context(
5959
if dp_world_size > 1 and forward_context.dp_metadata is not None:
6060
forward_context.max_tokens_across_dp = forward_context.dp_metadata.max_tokens_across_dp_cpu.item(
6161
)
62+
elif num_tokens is not None:
63+
forward_context.max_tokens_across_dp = num_tokens
6264
elif attn_metadata is not None:
63-
forward_context.max_tokens_across_dp = num_tokens or attn_metadata.num_actual_tokens
65+
if hasattr(attn_metadata, 'num_actual_tokens'):
66+
forward_context.max_tokens_across_dp = attn_metadata.num_actual_tokens
67+
else:
68+
forward_context.max_tokens_across_dp = attn_metadata.num_prefill_tokens + attn_metadata.num_decode_tokens
6469
else:
6570
forward_context.max_tokens_across_dp = None
6671

0 commit comments

Comments
 (0)