Include benchmark with `zarr.storage.ObjectStore`

[kylebarron]

👋 Hello, just learned about this cool project.

I see my collaborator @maxrjones already mentioned it here #1 (comment), but I'd encourage you to also test out the new zarr.storage.ObjectStore as a backend in addition to the default fsspec-based Zarr backend. This is new as of Zarr v3.0.7 (see zarr-developers/zarr-python#1661) and in https://github.com/maxrjones/zarr-obstore-performance we found it to have 3x faster wall time than the fsspec backend for loading a 24GB zarr array from S3.

If you do add that to your benchmarks I'll be really curious to see how it fares!

[dstansby]

Thanks for opening this, that sounds like a good idea! Is it only for remote data though? Currently we're just testing local storage, so it would be a bit more work to implement tests on remote stores.

[kylebarron]

Ah indeed the ObjectStore adapter is primarily for remote data. It's possible to use it also for local files but it's unclear whether that would be faster.