HDMF-Zarr 2025 Major Roadmap

[rly]

Currently, maintaining HDMF-Zarr as a separate package from PyNWB and HDMF is challenging for us and the broader community because of complex dependencies between the packages. I think HDMF-Zarr users would also benefit from not having to worry about installing another package on top of HDMF. To improve this issues, now that HDMF-Zarr is more stable, the development team is considering merging HDMF-Zarr into PyNWB and HDMF.

Specifically:

1. Do not release 1.0 version of HDMF-Zarr. Instead, make pre-1.0 releases.
2. Move the nwb module to https://github.com/NeurodataWithoutBorders/pynwb (Spring)
3. Merge the remaining modules into HDMF (Spring).
   • Users would install the HDMF-Zarr backend and its dependencies via pip install "hdmf[zarr]". We could later consider installing HDMF-Zarr by default.
4. Refactor common code between the HDF5 code and Zarr code (Summer)
5. Make major release of HDMF (Summer)
6. Declare this repo defunct and recommend users run pip install "hdmf[zarr]"

[h-mayorquin]

Just to keep track, where does zarr v3 support falls within this timeline (if it does)?

#202

[rly]

We will investigate and evaluate how to support Zarr v3. There are some challenges with strings and structured arrays. I expect that hdmf-zarr will support Zarr v3 in the first half of 2025.

[h-mayorquin]

Thanks, following the discussion here then:
zarr-developers/zarr-specs#325