v0.3.0

Highlights

🧠 New Models

We are excited to expand our model library with the addition of four powerful new architectures.

• 🎉 MesaNet by @sustcsonglin
• 🛣️ PaTH by @sustcsonglin
• 🐍 Comba by @AwesomeSeq @yzhangcs
• 🐳 MLA by @toothacher17 @yzhangcs

What's Changed

• [MesaNet] add kernel impl. by @sustcsonglin in #419
• [GDN] Add support for inference with GVA by @yzhangcs in #429
• [HGRN] remove unused q_conv1d by @yibozhong in #430
• Update mesa_net.py by @jovoswald in #434
• [Gated DeltaNet] Refactor the kernel to remove one matrix inversion by @sustcsonglin in #433
• [Modules] Add L2Warp to maintain bf16 precision by @zhiyuan1i in #438
• [RWKV]: Set default scale to None by @zhiyuan1i in #445
• [Typos] Change scale docs to (Optional[float]) [skip test] by @zhiyuan1i in #446
• [Modules] Enhance Testing of l2warp by @zhiyuan1i in #448
• [CI] Upgrade CI envs to torch~=2.7.0 by @zhiyuan1i in #450
• [Mesa] misc. fix by @sustcsonglin in #449
• [Models]: Add Comba Implementation by @AwesomeSeq in #444
• [Test] Walk around the bug of causal_conv1d by @zhiyuan1i in #453
• [Utils] Add deprecation handling for kwargs with deprecate_kwarg decorator by @yzhangcs in #455
• [ShortConv] Replace use_fast_conv1d with backend parameter by @yzhangcs in #456
• [Docs] Update tensor shape descriptions and deprecate head_first argument by @yzhangcs in #457
• [Simple GLA] Support dg when dht passed by @yzhangcs in #459
• [Mesa] Improve precision by @sustcsonglin in #460
• [Comba] Remove problematic safe_exp by @yzhangcs in #466
• [TokenShift] Fix invalid argument on AMD GPUs by @zhiyuan1i in #464
• [Test] Refractor model testing[skip test] by @zhiyuan1i in #467
• [Testing] Enhance generation testing by @sustcsonglin in #468
• [Simple GLA] Remove unnecessary dg for data-independent decay by @yzhangcs in #469
• [CI] Update workflow by @zhiyuan1i in #473
• [Misc.] Enhance support for some platforms by @zhiyuan1i in #470
• [Gated Delta Product] Optimize kernels by @sustcsonglin in #472
• [README] Add support for aarch64 by @zhiyuan1i in #475
• [Cache] Fix bad seen_tokens update by @yzhangcs in #478
• [CI] Revert causal-conv1d to 2a288a1 by @zhiyuan1i in #480
• [Parallel] Fix all tokens offsets by @yzhangcs in #479
• Use tl.exp2 for all gating operations by @yzhangcs in #361
• Refactor modeling tests by @yzhangcs in #482
• Add L2_norm for p in Recurrent ops to fix generation error by @AwesomeSeq in #483
• Refactor benchmark: adapt to latest FLA benchmark interface by @yuweih205 in #488
• [GLA] Remove all safe_exp ops by @yzhangcs in #489
• [MesaNet] Remove all safe_exp ops & Refactor tests by @yzhangcs in #490
• [Misc.] Support PT2.5 by @zhiyuan1i in #491
• [Misc.] Fast testing & Autotune by @sustcsonglin in #476
• fix: update import path for causal_conv1d by @yuweih205 in #492
• Make RWKV-7 init match official RWKV-LM by @johanwind in #493
• Modernize the fused_chunk impls by @yzhangcs in #437
• [ShortConv] Fix bad conv weight input shape during inference by @yzhangcs in #495
• [DeltaProduct] chore: remove unused functions by @timurcarstensen in #496
• [CI] Fix pipeline in GPU CIs by @zhiyuan1i in #497
• [RWKV] Make torch.compile decorator compatible with python3.10 by @zhiyuan1i in #498
• [GDN] Fuse 64x64 matrix inverse kernel by @yzhangcs in #501
• [L2Norm] Speedup by saving rstd by @yzhangcs in #506
• [Norm] Move eps out of sqrt by @yzhangcs in #508
• Correct types of constructor arguments with issues for configuration classes by @V0XNIHILI in #509
• Fix typo: suppoerted -> supported by @zxytim in #510
• [RWKV7] Increase Lora shape for headdim>64 by @zhiyuan1i in #512
• [Delta Rule] Support gk for WY reprs by @yzhangcs in #514
• [PaTH attention] Support headdim 128 & refactor kernel for better stability by @sustcsonglin in #503
• [Rotary] Fix max_seqlen under varlen mode by @yzhangcs in #516
• [Misc] Skip testing models on Nvidia 4090 CI by @zhiyuan1i in #517
• [GDP] Delete duplicated code by @yzhangcs in #518
• [WIP] Add MLA layers into fla by @toothacher17 in #395
• [Mamba] Add triton conv1d backend and fix mamba2 test by @zhiyuan1i in #520
• [Typo] Fix types in all configuration files[skip test] by @V0XNIHILI in #513
• [GSA] Fix memory boundary conditions by @JusenD in #527

New Contributors

• @jovoswald made their first contribution in #434
• @AwesomeSeq made their first contribution in #444
• @yuweih205 made their first contribution in #488
• @V0XNIHILI made their first contribution in #509
• @zxytim made their first contribution in #510
• @toothacher17 made their first contribution in #395
• @JusenD made their first contribution in #527

Full Changelog: v0.2.2...v0.3.0