Gemma3 v1.22 changes (Sliding_Window feature + few others) #1660
+167
−72
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FusedSDPA kernel with window_size+causal only works when seq_len is multiple of SLICE_SIZE. If not, fallback to the original implementation which creates attention_mask with window_size
-Move the seq_len check for use_sdpa_window to attn_metadata -Automatically set all environment variable
Additional changes from PR1597
remove print statement
For Sliding FusedSDPA kernel, add threshold variable to enable or disable.
jiminha
requested review from
kzawora-intel,
madamczyk-intel,
michalkuligowski,
mgawarkiewicz-intel,
vivekgoe,
afierka-intel,
xuechendi,
jikunshang,
mswiniarsk and
PatrykWo
as code owners
|
This has co-dependency with #1660 |
jiminha
changed the title
QKV with sliding kernel with causal is not supported yet. However when attn_mask is there, QKV is supported, so no need for repeat KV.
libinta
reviewed
Jul 29, 2025
Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>
Signed-off-by: Hongmin Fan <fanhongmin@google.com>
libinta
changed the title
Comment on lines
+356
to
+358
| self.slice_size = int(os.getenv("PT_HPU_SDPA_BC_FACTOR", "1024")) | ||
| self.sliding_window_thld = int( | ||
| os.environ.get('VLLM_FUSEDSDPA_SLIDE_THLD', '8192')) |
There was a problem hiding this comment.
are those two new env variables? shouldnt they be in readme?
There was a problem hiding this comment.
PT one is not new, it's related to internal flashattn3 implementation. For vllm, we try to give the optimized number ourselves for now.
Jianhong-Zhang
force-pushed
the
jha/slidingthld
branch
from
ffa1c79 to
d33c55c
Compare
Jianhong-Zhang
force-pushed
the
jha/slidingthld
branch
from
d33c55c to
8eda4aa
Compare
…position 1) must be Tensor, not NoneType
michalkuligowski
approved these changes
Jul 31, 2025
requirements/hpu.txt
Outdated
| @@ -7,4 +7,4 @@ ray | |||
| triton==3.1.0 | |||
| setuptools>=77.0.3 | |||
| setuptools-scm>=8 | |||
| vllm-hpu-extension @ git+https://github.com/HabanaAI/vllm-hpu-extension.git@61dafb3 | |||
| vllm-hpu-extension @ git+https://github.com/HabanaAI/vllm-hpu-extension.git@009adb2 | |||
There was a problem hiding this comment.
code looks good, please update sha to the right once HabanaAI/vllm-hpu-extension#315 is merged (it will differ from 009adb2 which is only visible on PR branch)
|
/run-gaudi-tests |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains following changes
upstream fixes
[Model][Gemma3] Cast image pixel values already on CPU vllm-project/vllm#18732
[Model] Fix a check for None but the return value was empty list in Gemma3 MM vision_embeddings vllm-project/vllm#21479
Mark invariant normalizer in Gemma as non-persistent vllm-project/vllm#19788
optimized Gemma3RMSNorm with FusedRMSNorm
Dependent on Fixes from PR#1635 applied to v1.22.0_next branch #1647
Run command with.
VLLM_FUSEDSDPA_SLIDE_THLD=2048 VLLM_EXPONENTIAL_BUCKETING=false VLLM_PROMPT_BS_BUCKET_MAX=64 VLLM_PROMPT_SEQ_BUCKET_STEP=1024 VLLM_PROMPT_SEQ_BUCKET_MAX=20480 PT_HPU_SDPA_QKV_SLICE_MODE_FWD=1