v0.5.0

This release refactors a large portion of the nnbench running and reporting functionality.

The singleton nnbench.BenchmarkRunner class was replaced with the nnbench.collect() and nnbench.run() APIs, which are more modular than the previous, largely stateless class, and allow for a more flexible setup of custom benchmarking scripts.

Also, SQLite and MLflow were added as new reporters for benchmark results, meaning you can now stream results to/from SQLite, and to an MLFlow instance. Read support for MLflow will be added in an upcoming release.

The nnbench command-line interface (CLI) now supports the nnbench compare subcommand, which can be used to display results of multiple benchmark runs in a single table, and optionally compare results with special functions called comparators. More information can be found in the documentation on the CLI.

What's Changed

• Add tests for nnbench.config by @nicholasjng in #187
• Add test coverage for nnbench.context by @nicholasjng in #188
• Fix dead nnbench logo link, remove stale TODO from quickstart by @nicholasjng in #190
• Run pre-commit autoupdate, rename decorator test file by @nicholasjng in #191
• packaging: Switch to PEP735 dependency groups by @nicholasjng in #192
• Replace nnbench.BenchmarkRunner with modular nnbench.collect() and nnbench.run() APIs by @nicholasjng in #193
• Bump setup-uv to v5, change Python install flow by @nicholasjng in #194
• Add zizmor pre-commit hook by @nicholasjng in #195
• Persist credentials for mike push jobs by @nicholasjng in #196
• refactor: Delete types submodule, inline everything into source file by @nicholasjng in #197
• Run pre-commit autoupdate by @nicholasjng in #198
• Use is instead of ==/!= operators in argument checks by @nicholasjng in #200
• Add BenchmarkFamily iterable by @nicholasjng in #199
• docs: Remove stale BenchmarkRunner references, update hooks by @nicholasjng in #201
• Remove nnbench.Memo and all related functionality by @nicholasjng in #202
• core: Discover benchmark families as well by @nicholasjng in #203
• docs: Allow writes for mike action by @nicholasjng in #204
• maint: Drop some dead code, update deps by @nicholasjng in #205
• feat(cli): Add -j option for running benchmarks in parallel by @nicholasjng in #206
• tests: Fix elapsed time calculation, use single delay import by @nicholasjng in #207
• fix: Use walk() to traverse more than one directory in benchmark collection by @nicholasjng in #208
• refactor: Renew file IO facilities by @nicholasjng in #210
• chore: Remove stale reporter code, switch package to use new IO protocol by @nicholasjng in #211
• chore: Upgrade lockfile dependencies, pre-commit hooks by @nicholasjng in #212
• Add MLFlowIO to enable MLflow logging of benchmark results by @nicholasjng in #213
• CLI: Restructure dispatch to be based on URI protocol by @nicholasjng in #214
• cli: Add -n switch to configure run name on command line by @nicholasjng in #216
• cli: Add --jsonifier switch to support user-given params JSONification by @nicholasjng in #217
• docs: Improve CLI reference and configuration guide by @nicholasjng in #218
• chore: Clean up several imports and definitions by @nicholasjng in #219
• cli: Allow provider names to be passed without "provider" key by @nicholasjng in #220
• refactor: Change date key and value type for benchmark timestamp by @nicholasjng in #221
• cli: Add query to reporter.read interface, add URI support to compare by @nicholasjng in #222
• feat: Add comparison class by @nicholasjng in #223
• chore: Rename BenchmarkRecord -> BenchmarkResult, add `(to|from)_reco… by @nicholasjng in #224
• feat: Add SQLite reporter by @nicholasjng in #225
• fix: Collapse results when loading them into comparison by @nicholasjng in #226
• fix: Set success to True by default to make boolean additions correct by @nicholasjng in #227
• pre-commit: Add zizmor config, update dependencies by @nicholasjng in #228
• refactor: Make reporter interfaces dynamic by @nicholasjng in #229
• refactor: Revert to single result writing in reporter by @nicholasjng in #231
• fix: Rename output path argument to "path", fix MLflow reporter by @nicholasjng in #232
• docs: Add dedicated doc on comparisons by @nicholasjng in #233
• chore: Add sqlite DBs to gitignore by @nicholasjng in #234
• cli: Remove -P parameter display option for nnbench compare by @nicholasjng in #235
• docs(cli): Add note on URIs for writing results by @nicholasjng in #236
• examples: Add ZenML pipeline example by @nicholasjng in #230

Full Changelog: v0.4.0...v0.5.0

v0.4.0

This release contains the nnbench CLI as a headline feature for running benchmarking workloads as part of scripts and CML pipelines.

It also contains a large refactor of core abstractions such as the benchmark runners and reporters. In particular, the nnbench.FileReporter got support for writing files with fsspec, making it possible to stream records to all major cloud storage providers among others.

What's Changed

• (fix) Change default == value to is by @maxmynter in #139
• Add nnbench video to readme by @maxmynter in #144
• Safe setting of CPU Info by @maxmynter in #146
• Bump oldest supported Python to 3.10 by @nicholasjng in #147
• Switch project to uv lockfile by @nicholasjng in #148
• Remove requirement files, strict version pins from examples by @nicholasjng in #149
• Remove unused io module stub by @nicholasjng in #150
• Add CLI entrypoints for module and program invocation by @nicholasjng in #152
• Run pre-commit autoupdate by @nicholasjng in #153
• Add Python 3.13 tests to CI by @nicholasjng in #155
• refactor: Remove context class, back to raw dictionaries by @nicholasjng in #154
• Remove setuptools_scm in favor of explicit version management by @nicholasjng in #156
• Remove transform facility and examples by @nicholasjng in #161
• Remove context processing options by @nicholasjng in #160
• Add Benchmark.to_list() and to_json() to improve record IO by @nicholasjng in #162
• Remove compression, support file reporting and context passing on the command line by @nicholasjng in #163
• Add nnbench compare subcommand, comparison function by @nicholasjng in #164
• Refactor console reporter by @nicholasjng in #165
• Add --version switch to print version by @nicholasjng in #166
• Inline boolean option pairs, change subcommand section by @nicholasjng in #167
• Add run name to benchmark record by @nicholasjng in #171
• Simplify JSON representation computation facility by @nicholasjng in #172
• Add a fixture facility to nnbench by @nicholasjng in #173
• Remove typechecks from benchmark runner by @nicholasjng in #174
• Add TOML config parsing stub by @nicholasjng in #175
• Add artifact outputs to gitignore by @nicholasjng in #176
• Add API docs for all direct nnbench submodules by @nicholasjng in #178
• Add nnbench CLI guide by @nicholasjng in #179
• Make arguments optional on ContextProviderDef by @nicholasjng in #181
• Add fixture usage guide document by @nicholasjng in #182
• Fix README lightning demo example by @nicholasjng in #183
• Update code examples, fix parameter hydration bug by @nicholasjng in #184
• Refactor jsonifier to be taken as an argument to BenchmarkRunner.run() by @nicholasjng in #185

Full Changelog: v0.3.0...v0.4.0

v0.3.0

This release contains the Memo class for better memory efficiency when benchmarking multiple models or datasets with large memory footprint in succession.

It also introduces Transforms as a way to encapsulate processing of records for multiple purposes, like generating statistics or serialization/deserialization.

What's Changed

• Example Guide: Benchmark on Saved Models by @maxmynter in #92
• Add a lightning demo to the README.md by @nicholasjng in #98
• Hotfix: Adjust snippet paths by @nicholasjng in #102
• Add Parameters to Benchmarks by @maxmynter in #103
• Run pre-commit autoupdate, bump dependencies by @nicholasjng in #104
• Implement a FSspec based ArtifactLoader by @maxmynter in #100
• Add Artifacts guide by @maxmynter in #107
• Remove the ArtifactCollection class by @maxmynter in #109
• Inline the _cleanup function for the FilePathArtifactLoader. by @maxmynter in #111
• Unique keys by @maxmynter in #112
• functools.partialize the nnbench.parametrize decorators. by @maxmynter in #116
• Add transform proposal by @nicholasjng in #113
• Make Transform an abstract base class, delete iapply method ... by @nicholasjng in #118
• Check loaded module names before importing a new one from a file by @nicholasjng in #119
• Rollforward of lakefs-spec doc changes by @nicholasjng in #123
• Add transform submodule, parameter compression transform by @nicholasjng in #124
• Feature: Thunks as an alternative to artifacts by @nicholasjng in #120
• Add new memoization guide by @nicholasjng in #129
• State injection into setup and teardown by @maxmynter in #127
• Add global memo cache and integrate with the setUp and teardown injection by @maxmynter in #130
• Update dev requirements, add Memray artifacts to gitignore by @nicholasjng in #136
• Organize nnbench.types.types into different files by @maxmynter in #135
• Implement memo garbage collection by @nicholasjng in #137
• Update HuggingFace example to use memoization by @nicholasjng in #133

Full Changelog: v0.2.0...v0.3.0

v0.2.0

Summary

This release makes a wide variety of benchmarking functionality available: Contexts for easier metadata organization, providers for basic information, file IO for record persistence on disk, and more.

It also adds new guides and code examples on how to benchmark code easily in a variety of scenarios, such as in orchestrator workflows and with duckDB.

What's Changed

• Add git info and Python package info providers by @nicholasjng in #64
• Group context structs under top-level semantic keys by @nicholasjng in #65
• Simplify nullcols helper, flatten nested context by @nicholasjng in #68
• Fix runner collection by checking file path first by @nicholasjng in #70
• Change nnbench.reporter file to submodule, add base reporter by @nicholasjng in #71
• Inline console reporter into BenchmarkReporter.display() by @nicholasjng in #76
• Orchestrator examples by @maxmynter in #73
• Improve some aspects on benchmark reporters by @nicholasjng in #77
• Add barebones duckDB reporter by @nicholasjng in #75
• Add BenchmarkContext Class by @maxmynter in #79
• Improve file drivers, add dict roundtrip methods to BenchmarkRecord by @nicholasjng in #82
• Unify batching in file drivers, remove read/write stubs on base reporter by @nicholasjng in #83
• Add compression algorithms facility to file IO by @nicholasjng in #84
• Streamlit Integration by @maxmynter in #81
• Add BQ example to showcase cloud streaming of benchmark records by @nicholasjng in #85
• Add parquet, ndjson drivers, change registration mechanisms by @nicholasjng in #87
• Revert file extension coercion, add raise statement for missing driver by @nicholasjng in #88
• Switch author and maintainer description in example .toml's by @maxmynter in #89
• Restructure test suite and add file IO roundtrip tests by @nicholasjng in #90
• Add duckDB example by @nicholasjng in #91
• Run pre-commit autoupdate, upgrade requirement lockfiles by @nicholasjng in #93

Full Changelog: v0.1.0...v0.2.0

v0.1.0

This marks the first stable release of nnbench. From now on, the package is available for installation on PyPI using standard Python dependency management tools like pip and poetry.

Contents

• Definition and parametrization of benchmarks via decorators
• Collection and running in a benchmark runner class
• Result reporting via a benchmark reporter class (currently, only tabular console output is supported.)
• Documentation including quickstart, a more advanced example, and an API reference.

Breaking changes

None.

Deprecations

None.

v0.1.0-rc1

This is a test PyPI release.