Skip to content

Anthropic Jan Jumprelu; triton spmm decoding; miscs #89

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 20 commits into from
Feb 11, 2025
Merged

Anthropic Jan Jumprelu; triton spmm decoding; miscs #89

merged 20 commits into from
Feb 11, 2025

Conversation

Hzfinfdu
Copy link
Member

This PR includes:

  • Implementing Anthropic Jan Jumprelu: init, schedule and JumpReLU Implementation.
  • triton spmm decoding: Faster decoding for fwd and bwd. Accelerates decoding when sparsity > 0.99
  • implement overriding dtype for cached activation

Hzfinfdu and others added 17 commits February 3, 2025 13:36
@Hzfinfdu
feat(anthropic jumprelu): initial implementation
3a51cb6
@Hzfinfdu
feat(anthropic jumprelu): training is ready
5ec09a6
@Hzfinfdu
feat(anthropic jumprelu): ready for training
5284ba5
@Hzfinfdu
fix(fp32 threshold): enforce jumprelu threshold uses fp32
de74592
@Hzfinfdu
merge jumprelu implementation with main
9c97b67
@Hzfinfdu
fix(fp32 threshold): enforce jumprelu threshold uses fp32
b6471ee
@Hzfinfdu
merge jumprelu implementation with main
98b0751
@Hzfinfdu
implement override dtype for cached activation
5b71a95
@Hzfinfdu
feat(spmm_decode): implement sparse mm decode few & bwd with triton
fd26455
@Hzfinfdu
fix(triton): add triton in pyproject
2175a57
@Hzfinfdu
fix(cached_activation): fix errors in implementation of overriding ca…
9750b49 
…ched activations
@Hzfinfdu
fix(tests;ruff): remove unused variables and functions. fix errors in…
63ec3cc 
… tests
@Hzfinfdu
fix(cc): fix cc inheriting logics of activation func
9e8e27f
@Frankstein73
fix(sae): fix type error
718c605
@Frankstein73
fix(kernel): fix type error
f6adefa
@Hzfinfdu
fixconda activate base
9356b09
@Hzfinfdu
Merge branch 'implement_jan_jumprelu' of github.com:OpenMOSS/Language…
ebb2c0f 
…-Model-SAEs into implement_jan_jumprelu
@Hzfinfdu Hzfinfdu requested review from dest1n1s and Frankstein73 and removed request for dest1n1s February 11, 2025 08:24
@dest1n1s dest1n1s merged commit 83acd31 into main Feb 11, 2025
2 checks passed
@Hzfinfdu Hzfinfdu deleted the implement_jan_jumprelu branch February 19, 2025 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dest1n1s dest1n1s dest1n1s approved these changes

@Frankstein73 Frankstein73 Awaiting requested review from Frankstein73

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Hzfinfdu @dest1n1s @Frankstein73