-
Notifications
You must be signed in to change notification settings - Fork 183
[Transform] Spinquant with R1 and R2 #1615
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
brian-dellabetta
wants to merge
38
commits into
main
Choose a base branch
from
bdellabe/transform-modifier
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 1 commit
Commits
Show all changes
38 commits
Select commit
Hold shift + click to select a range
ba617db
wip
kylesayrs 2f5b1c8
use random-hadamard, add correctness tests
kylesayrs 3aa35e7
add correctness test, note that precision makes a large difference
kylesayrs b6c088e
add on lifecycle methods
brian-dellabetta d1eb2a1
Merge branch 'main' into kylesayrs/transform-modifier
brian-dellabetta 3207124
TransformModifier with SpinQuant R1&R2
brian-dellabetta a88ca3c
spinquant and quip_online, running but outputting gibberish
brian-dellabetta 5bd51df
updated example
brian-dellabetta 3c216dd
DummyModel script
brian-dellabetta bbcdc8c
implement fuse_norm_linears
kylesayrs bd7f4d5
Merge branch 'kylesayrs/fuse-helpers' into bdellabe/transform-modifier
kylesayrs f5c2150
R1 working
kylesayrs dc5c30c
add r2, increase precision
kylesayrs 7172c26
spinquant modifier
kylesayrs 9298e82
remove space
kylesayrs f77226d
use iterable
kylesayrs fdb64b5
add rotation validation
kylesayrs 5daa2d5
embedding fusion
kylesayrs 0e9af7b
add missing norm fusion
kylesayrs fce83be
use norm mappings
kylesayrs a979f8a
break into separate files
kylesayrs 4cab29e
small cleanup
kylesayrs f1cc987
cleanup
kylesayrs a7bb2e2
more cleanup
kylesayrs 0cf0188
make new weight on cpu
kylesayrs 53ea307
standardize, make modifier serializable
kylesayrs 4b4257f
add compress model script
kylesayrs dc7ac1a
use untie_word_embeddings
kylesayrs 8542f8d
style
kylesayrs b1e637e
better registery logic
kylesayrs b44ac81
remove dummy model test (add later)
kylesayrs 7a52b71
docstring
kylesayrs f4d7ec6
update docstring
kylesayrs f18d0e8
rename example file
kylesayrs cec2914
use match_modules_set
kylesayrs f6c797e
Merge branch 'main' into bdellabe/transform-modifier
brian-dellabetta 0c5c514
unit test fixes
brian-dellabetta f2ef7cf
style fixes
brian-dellabetta File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kylesayrs why are we excluding these? wouldn't we want them to persist in json?