v0.5.0

This release refactors a large portion of the nnbench running and reporting functionality.

The singleton nnbench.BenchmarkRunner class was replaced with the nnbench.collect() and nnbench.run() APIs, which are more modular than the previous, largely stateless class, and allow for a more flexible setup of custom benchmarking scripts.

Also, SQLite and MLflow were added as new reporters for benchmark results, meaning you can now stream results to/from SQLite, and to an MLFlow instance. Read support for MLflow will be added in an upcoming release.

The nnbench command-line interface (CLI) now supports the nnbench compare subcommand, which can be used to display results of multiple benchmark runs in a single table, and optionally compare results with special functions called comparators. More information can be found in the documentation on the CLI.

What's Changed

• Add tests for nnbench.config by @nicholasjng in #187
• Add test coverage for nnbench.context by @nicholasjng in #188
• Fix dead nnbench logo link, remove stale TODO from quickstart by @nicholasjng in #190
• Run pre-commit autoupdate, rename decorator test file by @nicholasjng in #191
• packaging: Switch to PEP735 dependency groups by @nicholasjng in #192
• Replace nnbench.BenchmarkRunner with modular nnbench.collect() and nnbench.run() APIs by @nicholasjng in #193
• Bump setup-uv to v5, change Python install flow by @nicholasjng in #194
• Add zizmor pre-commit hook by @nicholasjng in #195
• Persist credentials for mike push jobs by @nicholasjng in #196
• refactor: Delete types submodule, inline everything into source file by @nicholasjng in #197
• Run pre-commit autoupdate by @nicholasjng in #198
• Use is instead of ==/!= operators in argument checks by @nicholasjng in #200
• Add BenchmarkFamily iterable by @nicholasjng in #199
• docs: Remove stale BenchmarkRunner references, update hooks by @nicholasjng in #201
• Remove nnbench.Memo and all related functionality by @nicholasjng in #202
• core: Discover benchmark families as well by @nicholasjng in #203
• docs: Allow writes for mike action by @nicholasjng in #204
• maint: Drop some dead code, update deps by @nicholasjng in #205
• feat(cli): Add -j option for running benchmarks in parallel by @nicholasjng in #206
• tests: Fix elapsed time calculation, use single delay import by @nicholasjng in #207
• fix: Use walk() to traverse more than one directory in benchmark collection by @nicholasjng in #208
• refactor: Renew file IO facilities by @nicholasjng in #210
• chore: Remove stale reporter code, switch package to use new IO protocol by @nicholasjng in #211
• chore: Upgrade lockfile dependencies, pre-commit hooks by @nicholasjng in #212
• Add MLFlowIO to enable MLflow logging of benchmark results by @nicholasjng in #213
• CLI: Restructure dispatch to be based on URI protocol by @nicholasjng in #214
• cli: Add -n switch to configure run name on command line by @nicholasjng in #216
• cli: Add --jsonifier switch to support user-given params JSONification by @nicholasjng in #217
• docs: Improve CLI reference and configuration guide by @nicholasjng in #218
• chore: Clean up several imports and definitions by @nicholasjng in #219
• cli: Allow provider names to be passed without "provider" key by @nicholasjng in #220
• refactor: Change date key and value type for benchmark timestamp by @nicholasjng in #221
• cli: Add query to reporter.read interface, add URI support to compare by @nicholasjng in #222
• feat: Add comparison class by @nicholasjng in #223
• chore: Rename BenchmarkRecord -> BenchmarkResult, add `(to|from)_reco… by @nicholasjng in #224
• feat: Add SQLite reporter by @nicholasjng in #225
• fix: Collapse results when loading them into comparison by @nicholasjng in #226
• fix: Set success to True by default to make boolean additions correct by @nicholasjng in #227
• pre-commit: Add zizmor config, update dependencies by @nicholasjng in #228
• refactor: Make reporter interfaces dynamic by @nicholasjng in #229
• refactor: Revert to single result writing in reporter by @nicholasjng in #231
• fix: Rename output path argument to "path", fix MLflow reporter by @nicholasjng in #232
• docs: Add dedicated doc on comparisons by @nicholasjng in #233
• chore: Add sqlite DBs to gitignore by @nicholasjng in #234
• cli: Remove -P parameter display option for nnbench compare by @nicholasjng in #235
• docs(cli): Add note on URIs for writing results by @nicholasjng in #236
• examples: Add ZenML pipeline example by @nicholasjng in #230

Full Changelog: v0.4.0...v0.5.0