|
| 1 | +# Attention Gym - Claude Context |
| 2 | + |
| 3 | +## Repository Overview |
| 4 | + |
| 5 | +Attention Gym is a collection of helpful tools and examples for working with PyTorch's FlexAttention API. This repository provides implementations of various attention mechanisms, performance benchmarks, and utility functions for researchers and developers. |
| 6 | + |
| 7 | +## Project Structure |
| 8 | + |
| 9 | +- `attn_gym/`: Main package containing attention implementations |
| 10 | + - `masks/`: Examples for creating BlockMasks (causal, sliding window, document masks, etc.) |
| 11 | + - `mods/`: Examples for creating score_mods (alibi, latent attention, softcapping) |
| 12 | + - `paged_attention/`: PagedAttention implementations and benchmarks |
| 13 | +- `examples/`: End-to-end implementations and benchmarks |
| 14 | +- `test/`: Test files for the project |
| 15 | + |
| 16 | +## Development Commands |
| 17 | + |
| 18 | +### Setup |
| 19 | +```bash |
| 20 | +# Install in development mode |
| 21 | +pip install -e ".[dev]" |
| 22 | + |
| 23 | +# Install pre-commit hooks |
| 24 | +pre-commit install |
| 25 | +``` |
| 26 | + |
| 27 | +### Testing |
| 28 | +```bash |
| 29 | +# Run tests |
| 30 | +pytest |
| 31 | + |
| 32 | +# Run specific test files |
| 33 | +pytest test/test_mods.py |
| 34 | +pytest test/test_natten.py |
| 35 | +``` |
| 36 | + |
| 37 | +### Code Quality |
| 38 | +```bash |
| 39 | +# Run linting |
| 40 | +ruff check |
| 41 | + |
| 42 | +# Run formatting |
| 43 | +ruff format |
| 44 | + |
| 45 | +# Run pre-commit on all files |
| 46 | +pre-commit run --all-files |
| 47 | +``` |
| 48 | + |
| 49 | +### Running Examples |
| 50 | +```bash |
| 51 | +# Most files can be executed directly |
| 52 | +python attn_gym/masks/document_mask.py |
| 53 | +python examples/benchmark.py |
| 54 | +python examples/mla.py |
| 55 | +``` |
| 56 | + |
| 57 | +## Key Dependencies |
| 58 | + |
| 59 | +- PyTorch (>=2.5 for FlexAttention support) |
| 60 | +- Optional: matplotlib, numpy (for visualization examples) |
| 61 | +- Dev: pytest, ruff, pre-commit |
| 62 | + |
| 63 | +## Package Information |
| 64 | + |
| 65 | +- Package name: `attn_gym` |
| 66 | +- Python requirement: >=3.9 |
| 67 | +- License: BSD 3-Clause |
| 68 | +- Line length: 99 characters (ruff config) |
0 commit comments