🎉 Flash-DMA v0.1.0 - Initial Release

[LoserCheems]

We're excited to announce the first official release of Flash-DMA (Flash Dynamic Mask Attention)!

🚀 What is Flash-DMA?

Flash-DMA is a high-performance attention implementation that combines:

• Flash Attention's memory efficiency
• Dynamic Mask Attention's sparse computation
• Support for extremely long sequences (128K+ tokens)

✨ Key Features

🔥 Performance

• Sparse Attention: Reduces computation from O(N²) to O(N·w) where w ≪ N
• Memory Efficient: Maintains O(N) memory complexity
• CUDA Accelerated: Custom sparse GEMM operations at kernel level

🛠️ Multiple Backends

• CUDA Backend: Maximum performance with custom kernels
• Triton Backend: Flexibility for research and development
• Flex Backend: Integration with Transformers library

📏 Long Sequence Support

• Efficiently handles sequences of 128K+ tokens
• Dynamic masking when sequence length exceeds keep_window_size
• Optimized memory layouts for large-scale processing

📦 Installation

Prerequisites

• Python 3.9+
• PyTorch 2.0+
• CUDA 11.8+
• NVIDIA GPU with Compute Capability 8.0+

Install from Source

git clone https://github.com/SmallDoges/flash-dmattn.git
cd flash-dmattn
git submodule update --init --recursive
pip install .

What's Changed

• Workspace by @LoserCheems in Workspace #1
• Add namespace_config to csrc by @LoserCheems in Add namespace_config to csrc #2
• Add hardware_info to csrc by @LoserCheems in Add hardware_info to csrc #3
• Add block_info to csrc by @LoserCheems in Add block_info to csrc #4
• Add flash params to csrc by @LoserCheems in Add flash params to csrc #5
• Workspace by @LoserCheems in Workspace #6
• Update golobal to Shared Memory operation by @LoserCheems in Update golobal to Shared Memory operation #7
• Update PREDICATES by @LoserCheems in Update PREDICATES #8
• Fix some nits for layout by @LoserCheems in Fix some nits for layout #9
• Workspace by @LoserCheems in Workspace #10
• Fix Dynamic Mask Attention Integration in FlashAttention CUDA Kernel by @Copilot in Fix Dynamic Mask Attention Integration in FlashAttention CUDA Kernel #12
• Fix dynamic mask attention equivalence issue between Python and CUDA by @Copilot in Fix dynamic mask attention equivalence issue between Python and CUDA #14
• Fix CUDA dynamic mask attention scaling to match Python implementation by @Copilot in Fix CUDA dynamic mask attention scaling to match Python implementation #16
• Update mask.h by @Evanwu1125 in Update mask.h #17
• Comprehensive README improvement with installation, usage examples, and documentation by @Copilot in Comprehensive README improvement with installation, usage examples, and documentation #19
• Adds no-topk variant for kernel performance analysis by @LoserCheems in Adds no-topk variant for kernel performance analysis #20
• Optimize sparse GEMM and enable in attention computation by @LoserCheems in Optimize sparse GEMM and enable in attention computation #21
• Adds row stride support to offset calculation methods by @LoserCheems in Adds row stride support to offset calculation methods #22
• Corrects ZOH tensor dimension comment by @LoserCheems in Corrects ZOH tensor dimension comment #23
• Adds rotary positional encoding operations by @LoserCheems in Adds rotary positional encoding operations #24
• Adds conditional softcap switch macro by @LoserCheems in Adds conditional softcap switch macro #25
• Removes unused template parameter from DynamicMask by @LoserCheems in Removes unused template parameter from DynamicMask #26
• Updates tensor offset calculations and formatting by @LoserCheems in Updates tensor offset calculations and formatting #27
• Adds split-K attention kernel with sparse computation by @LoserCheems in Adds split-K attention kernel with sparse computation #28
• Adds dropout support and softcap feature to flash attention by @LoserCheems in Adds dropout support and softcap feature to flash attention #29
• Add specialized CUDA kernels for multi-head attention with various head dimensions by @LoserCheems in Add specialized CUDA kernels for multi-head attention with various head dimensions #30
• Remove cub submodule and add cutlass; implement FlashDynamicMaskAttention by @LoserCheems in Remove cub submodule and add cutlass; implement FlashDynamicMaskAttention #31
• Refactors setup.py for production-ready package build by @LoserCheems in Refactors setup.py for production-ready package build #32
• Update integration by @LoserCheems in Update integration #33
• Adds comprehensive API reference documentation by @LoserCheems in Adds comprehensive API reference documentation #34
• Updates README with improved technical accuracy and examples by @LoserCheems in Updates README with improved technical accuracy and examples #35
• Fix bug by @LoserCheems in Fix bug #36
• Adds column stride parameters to ZOH_params struct by @LoserCheems in Adds column stride parameters to ZOH_params struct #37
• Adds column stride support to offset calculations by @LoserCheems in Adds column stride support to offset calculations #38
• Fixes attention benchmarking and expands test coverage by @LoserCheems in Fixes attention benchmarking and expands test coverage #39
• Reorders stride parameter assignments by @LoserCheems in Reorders stride parameter assignments #40
• Adds column stride support to tensor memory layouts by @LoserCheems in Adds column stride support to tensor memory layouts #41
• Improves code clarity and test coverage by @LoserCheems in Improves code clarity and test coverage #42
• Updates copy function defaults and clarifies comments by @LoserCheems in Updates copy function defaults and clarifies comments #43
• Updates copy operations to use improved vectorization by @LoserCheems in Updates copy operations to use improved vectorization #44
• Updates benchmark test configurations for better coverage by @LoserCheems in Updates benchmark test configurations for better coverage #45
• Temporarily disables Split-KV feature by @LoserCheems in Temporarily disables Split-KV feature #46
• Optimizes CUDA kernel block sizes for better occupancy by @LoserCheems in Optimizes CUDA kernel block sizes for better occupancy #49
• Enables test case for 512x512 input dimensions by @LoserCheems in Enables test case for 512x512 input dimensions #50
• Renames dzero_hold to dzoh and adds column stride by @LoserCheems in Renames dzero_hold to dzoh and adds column stride #51
• Improves code formatting consistency in comments by @LoserCheems in Improves code formatting consistency in comments #52
• Fixes tensor addressing for ZOH and active mask in splitkv by @LoserCheems in Fixes tensor addressing for ZOH and active mask in splitkv #53
• Refactor attention mask and bias structures for clarity by @LoserCheems in Refactor attention mask and bias structures for clarity #54
• Refactor backward kernel for attention mask and bias support by @LoserCheems in Refactor backward kernel for attention mask and bias support #55
• Adds Flash Attention implementation with dynamic masking by @LoserCheems in Adds Flash Attention implementation with dynamic masking #56
• Fixes mask validation in forward kernel by @LoserCheems in Fixes mask validation in forward kernel #57
• Fixes mask comparison and scaling logic in attention kernel by @LoserCheems in Fixes mask comparison and scaling logic in attention kernel #58
• Enhance Flash Attention with required parameters and improved backward pass by @LoserCheems in Enhance Flash Attention with required parameters and improved backward pass #59
• Reorganizes flash attention files into instantiations directory by @LoserCheems in Reorganizes flash attention files into instantiations directory #60
• Rename flash_dma to flash_dmattn and improve usability by @LoserCheems in Rename flash_dma to flash_dmattn and improve usability #61
• Adds bias gradient computation to backward kernel by @LoserCheems in Adds bias gradient computation to backward kernel #62
• Add backend selection and dynamic mask attention support by @LoserCheems in Add backend selection and dynamic mask attention support #63
• Update by @LoserCheems in Update #64
• Removes no-topk CUDA implementation from benchmarks by @LoserCheems in Removes no-topk CUDA implementation from benchmarks #65
• Enables comprehensive benchmark configurations by @LoserCheems in Enables comprehensive benchmark configurations #66
• Renames Flash Attention to SDPA in benchmark suite by @LoserCheems in Renames Flash Attention to SDPA in benchmark suite #67
• Refactors variable declarations for better readability by @LoserCheems in Refactors variable declarations for better readability #68
• Add bias gradient computation support in backward kernel by @LoserCheems in Add bias gradient computation support in backward kernel #69
• Fix function naming and standardize memory copy alignment in attention kernel by @LoserCheems in Fix function naming and standardize memory copy alignment in attention kernel #70
• Adds unified mask application function with causal support by @LoserCheems in Adds unified mask application function with causal support #71
• Enables Split-KV avoidance and updates error messages by @LoserCheems in Enables Split-KV avoidance and updates error messages #74
• Adds variable length forward pass support by @LoserCheems in Adds variable length forward pass support #75
• Simplify attention mask and bias parameter naming by @LoserCheems in Simplify attention mask and bias parameter naming #76
• Remove unused parameters and simplify mask logic by @LoserCheems in Remove unused parameters and simplify mask logic #77
• Add CUDA-integrated flash attention interface by @LoserCheems in Add CUDA-integrated flash attention interface #78
• Improves version comparison using packaging library by @LoserCheems in Improves version comparison using packaging library #79
• Refactor CUDA interface for improved usability by @LoserCheems in Refactor CUDA interface for improved usability #80
• Refactors CUDA implementation to use new interface by @LoserCheems in Refactors CUDA implementation to use new interface #81
• Refactors dynamic mask function to improve clarity by @LoserCheems in Refactors dynamic mask function to improve clarity #82
• Expands API documentation with comprehensive interface guide by @LoserCheems in Expands API documentation with comprehensive interface guide #83
• Refactors API to use unified flash attention interface by @LoserCheems in Refactors API to use unified flash attention interface #84
• Convert banner image from JPG to PNG format by @LoserCheems in Convert banner image from JPG to PNG format #85
• Updates flash attention banner image by @LoserCheems in Updates flash attention banner image #86
• Updates citation format and adds acknowledgment by @LoserCheems in Updates citation format and adds acknowledgment #87
• Adds comprehensive documentation with logo and Chinese translation by @LoserCheems in Adds comprehensive documentation with logo and Chinese translation #89
• Simplify and standardize flex attention interface by @LoserCheems in Simplify and standardize flex attention interface #90
• Standardize parameter naming and improve API consistency in attention functions by @LoserCheems in Standardize parameter naming and improve API consistency in attention functions #91
• Implement Doge model with dynamic attention mechanism by @LoserCheems in Implement Doge model with dynamic attention mechanism #92
• Remove dropout functionality from flash attention by @LoserCheems in Remove dropout functionality from flash attention #93
• Make attention parameters optional with defaults and simplify API documentation by @LoserCheems in Make attention parameters optional with defaults and simplify API documentation #95
• Improves API parameter naming consistency by @LoserCheems in Improves API parameter naming consistency #96
• Adds GitHub issue templates and PR template by @LoserCheems in Adds GitHub issue templates and PR template #97
• Streamlines benchmark suite structure and test scope by @LoserCheems in Streamlines benchmark suite structure and test scope #99
• Update contributor list, add citation metadata, and enhance documentation by @LoserCheems in Update contributor list, add citation metadata, and enhance documentation #100
• Remove paper citation and author information from README files by @LoserCheems in Remove paper citation and author information from README files #102

New Contributors

• @LoserCheems made their first contribution in Workspace #1
• @Copilot made their first contribution in Fix Dynamic Mask Attention Integration in FlashAttention CUDA Kernel #12
• @Evanwu1125 made their first contribution in Update mask.h #17

Full Changelog: https://github.com/SmallDoges/flash-dmattn/commits/0.1.0

---

This discussion was created from the release 🎉 Flash-DMA v0.1.0 - Initial Release.

[LoserCheems]

LGTM🤗