Skip to content

[WIP] [Transform] q_attn and k_cache locations #334

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[WIP] [Transform] q_attn and k_cache locations #334

wants to merge 1 commit into from
+75 −0

Conversation

kylesayrs
Copy link
Contributor

@kylesayrs kylesayrs commented May 31, 2025 •
edited
Loading

Because attention is very standardized in transformers via the AttentionInterface, this provides a convenient way to hook into attention and apply transforms

@kylesayrs kylesayrs changed the base branch from main to kylesayrs/transform_factory May 31, 2025 05:49
Base automatically changed from kylesayrs/transform_factory to main June 10, 2025 15:24
@kylesayrs
wip concept
da36ca6 
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
@kylesayrs kylesayrs force-pushed the kylesayrs/transform_attn_locations branch from 96dee4e to da36ca6 Compare June 12, 2025 04:23
dsikka
dsikka reviewed Jul 3, 2025
View reviewed changes
Copy link
Collaborator

@dsikka dsikka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this ready for review?

@kylesayrs
Copy link
Contributor Author

I prefer this
https://github.com/vllm-project/llm-compressor/blob/7f8366706b7f22e51c9f2d9a50881b7cdd892617/src/llmcompressor/modifiers/quantization/attention.py

@kylesayrs kylesayrs closed this Jul 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dsikka dsikka dsikka left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kylesayrs @dsikka