v0.3.0
Highlights
π§ New Models
We are excited to expand our model library with the addition of four powerful new architectures.
- π MesaNet by @sustcsonglin
- π£οΈ PaTH by @sustcsonglin
- π Comba by @AwesomeSeq @yzhangcs
- π³ MLA by @toothacher17 @yzhangcs
What's Changed
- [MesaNet] add kernel impl. by @sustcsonglin in #419
- [GDN] Add support for inference with GVA by @yzhangcs in #429
- [HGRN] remove unused q_conv1d by @yibozhong in #430
- Update mesa_net.py by @jovoswald in #434
- [Gated DeltaNet] Refactor the kernel to remove one matrix inversion by @sustcsonglin in #433
- [Modules] Add
L2Warpto maintain bf16 precision by @zhiyuan1i in #438 - [RWKV]: Set default scale to None by @zhiyuan1i in #445
- [Typos] Change scale docs to (Optional[float]) [skip test] by @zhiyuan1i in #446
- [Modules] Enhance Testing of
l2warpby @zhiyuan1i in #448 - [CI] Upgrade CI envs to torch~=2.7.0 by @zhiyuan1i in #450
- [Mesa] misc. fix by @sustcsonglin in #449
- [Models]: Add Comba Implementation by @AwesomeSeq in #444
- [Test] Walk around the bug of
causal_conv1dby @zhiyuan1i in #453 - [Utils] Add deprecation handling for kwargs with
deprecate_kwargdecorator by @yzhangcs in #455 - [ShortConv] Replace
use_fast_conv1dwithbackendparameter by @yzhangcs in #456 - [Docs] Update tensor shape descriptions and deprecate
head_firstargument by @yzhangcs in #457 - [Simple GLA] Support dg when dht passed by @yzhangcs in #459
- [Mesa] Improve precision by @sustcsonglin in #460
- [Comba] Remove problematic
safe_expby @yzhangcs in #466 - [TokenShift] Fix invalid argument on AMD GPUs by @zhiyuan1i in #464
- [Test] Refractor model testing[skip test] by @zhiyuan1i in #467
- [Testing] Enhance generation testing by @sustcsonglin in #468
- [Simple GLA] Remove unnecessary dg for data-independent decay by @yzhangcs in #469
- [CI] Update workflow by @zhiyuan1i in #473
- [Misc.] Enhance support for some platforms by @zhiyuan1i in #470
- [Gated Delta Product] Optimize kernels by @sustcsonglin in #472
- [README] Add support for aarch64 by @zhiyuan1i in #475
- [Cache] Fix bad
seen_tokensupdate by @yzhangcs in #478 - [CI] Revert causal-conv1d to
2a288a1by @zhiyuan1i in #480 - [Parallel] Fix all tokens offsets by @yzhangcs in #479
- Use
tl.exp2for all gating operations by @yzhangcs in #361 - Refactor modeling tests by @yzhangcs in #482
- Add L2_norm for p in Recurrent ops to fix generation error by @AwesomeSeq in #483
- Refactor benchmark: adapt to latest FLA benchmark interface by @yuweih205 in #488
- [GLA] Remove all
safe_expops by @yzhangcs in #489 - [MesaNet] Remove all
safe_expops & Refactor tests by @yzhangcs in #490 - [Misc.] Support PT2.5 by @zhiyuan1i in #491
- [Misc.] Fast testing & Autotune by @sustcsonglin in #476
- fix: update import path for causal_conv1d by @yuweih205 in #492
- Make RWKV-7 init match official RWKV-LM by @johanwind in #493
- Modernize the
fused_chunkimpls by @yzhangcs in #437 - [ShortConv] Fix bad conv weight input shape during inference by @yzhangcs in #495
- [DeltaProduct] chore: remove unused functions by @timurcarstensen in #496
- [CI] Fix pipeline in GPU CIs by @zhiyuan1i in #497
- [RWKV] Make
torch.compiledecorator compatible with python3.10 by @zhiyuan1i in #498 - [GDN] Fuse 64x64 matrix inverse kernel by @yzhangcs in #501
- [L2Norm] Speedup by saving rstd by @yzhangcs in #506
- [Norm] Move eps out of sqrt by @yzhangcs in #508
- Correct types of constructor arguments with issues for configuration classes by @V0XNIHILI in #509
- Fix typo: suppoerted -> supported by @zxytim in #510
- [RWKV7] Increase Lora shape for headdim>64 by @zhiyuan1i in #512
- [Delta Rule] Support gk for WY reprs by @yzhangcs in #514
- [PaTH attention] Support headdim 128 & refactor kernel for better stability by @sustcsonglin in #503
- [Rotary] Fix
max_seqlenunder varlen mode by @yzhangcs in #516 - [Misc] Skip testing models on Nvidia 4090 CI by @zhiyuan1i in #517
- [GDP] Delete duplicated code by @yzhangcs in #518
- [WIP] Add MLA layers into fla by @toothacher17 in #395
- [Mamba] Add triton conv1d backend and fix mamba2 test by @zhiyuan1i in #520
- [Typo] Fix types in all configuration files[skip test] by @V0XNIHILI in #513
- [GSA] Fix memory boundary conditions by @JusenD in #527
New Contributors
- @jovoswald made their first contribution in #434
- @AwesomeSeq made their first contribution in #444
- @yuweih205 made their first contribution in #488
- @V0XNIHILI made their first contribution in #509
- @zxytim made their first contribution in #510
- @toothacher17 made their first contribution in #395
- @JusenD made their first contribution in #527
Full Changelog: v0.2.2...v0.3.0