[Bug]:

[rcpeene]

What happened?

I am trying to open a zarr nwb in order to append CCF information to the electrodes table, but it fails to open.

Note that .zmetadata is present in the Zarr NWB file. It's worth mentioning that I generated this hdmf-zarr==0.8.0 but and now trying to open it with latest (0.10.0), perhaps that is causing this.

Steps to Reproduce

Run the following code snippet, where there is a Zarr NWB present at `scratch_nwb_path`


    print('Reading NWB in append mode:', scratch_nwb_path)
    print(os.listdir(scratch_folder))
    if scratch_nwb_path.is_dir():
        print(os.listdir(scratch_nwb_path))
    with NWBZarrIO(str(scratch_nwb_path), mode='a') as read_io:
        nwb = read_io.read()

Traceback

Reading NWB in append mode: ../scratch/ecephys_747816_2024-08-14_13-57-37_experiment1_recording1.nwb
['ecephys_747816_2024-08-14_13-57-37_experiment1_recording1.nwb']
['file_create_date', 'intervals', 'analysis', 'specifications', 'stimulus', '.zattrs', 'general', 'acquisition', '.zmetadata', '.zgroup', 'processing', 'timestamps_reference_time', 'identifier', 'units', 'session_description', 'session_start_time']
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 566, in __open_file_consolidated
    return zarr.open_consolidated(
  File "/opt/conda/lib/python3.9/site-packages/zarr/convenience.py", line 1360, in open_consolidated
    meta_store = ConsolidatedStoreClass(store, metadata_key=metadata_key)
  File "/opt/conda/lib/python3.9/site-packages/zarr/storage.py", line 3046, in __init__
    meta = json_loads(self.store[metadata_key])
  File "/opt/conda/lib/python3.9/site-packages/zarr/storage.py", line 1120, in __getitem__
    raise KeyError(key)
KeyError: '.zmetadata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/capsule/code/run_capsule.py", line 140, in <module>
    if __name__ == "__main__": run()
  File "/root/capsule/code/run_capsule.py", line 124, in run
    nwb = read_io.read()
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/backends/io.py", line 56, in read
    f_builder = self.read_builder()
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 1470, in read_builder
    f_builder = self.__read_group(self.__file, ROOT_NAME)
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 1548, in __read_group
    sub_builder = self.__read_group(sub_group, sub_name)
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 1548, in __read_group
    sub_builder = self.__read_group(sub_group, sub_name)
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 1557, in __read_group
    self.__read_links(zarr_obj=zarr_obj, parent=ret)
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 1576, in __read_links
    target_name, target_zarr_obj = self.resolve_ref(link)
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 822, in resolve_ref
    target_zarr_obj = self.__open_file_consolidated(
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 573, in __open_file_consolidated
    return zarr.open(
  File "/opt/conda/lib/python3.9/site-packages/zarr/convenience.py", line 137, in open
    raise PathNotFoundError(path)
zarr.errors.PathNotFoundError: nothing found at path ''
(base) root@a09d29db9803:~/capsule/code# 

### Operating System

Windows

### Python Version

3.9

### Package Versions

_No response_

[alejoe91]

Getting the same error. Actually, in my case this also happens with a round-trip test, where an NWB-zarr file is generated and then read again with hdmf-zarr 0.10.0

See PR here: SpikeInterface/spikeinterface#3598

[rly]

@rcpeene @alejoe91 Thank you for the report. We will take a look and also make sure we include an appropriate round-trip test to catch errors like this in the future.

[rly]

The error appears to come from reading a reference from a zarr store that is not the current working directory. Below is a minimal working example. It fails when reading from "test/test.nwb" but succeeds when reading from "test.nwb". I will continue to investigate.

from pynwb.testing.mock.file import mock_NWBFile
from pynwb.testing.mock.device import mock_Device
from pynwb.testing.mock.ecephys import mock_ElectricalSeries, mock_ElectrodeGroup
import numpy as np

from hdmf_zarr import NWBZarrIO

nwbfile = mock_NWBFile()
device = mock_Device(name="probe")
nwbfile.add_device(device)
nwbfile.add_electrode_column(name="channel_name", description="channel name")

electrode_group = mock_ElectrodeGroup(device=device)
nwbfile.add_electrode_group(electrode_group)
nwbfile.add_electrode(group=electrode_group, location="unknown", channel_name="channel_name1")

electrode_region = nwbfile.create_electrode_table_region(region=[0], description="electrodes")
electrical_series = mock_ElectricalSeries(
    name="electrical_series_name",
    electrodes=electrode_region,
    data=np.array([1, 2]),
    rate=1000.0,
)
nwbfile.add_acquisition(electrical_series)

import os
os.makedirs("test", exist_ok=True)
with NWBZarrIO("test/test.nwb", mode="w") as io:
    io.write(nwbfile)

io = NWBZarrIO("test/test.nwb", mode="r")
read_nwbfile = io.read()
io.close()

Error:

File ~/miniconda3/envs/temp/lib/python3.12/site-packages/zarr/convenience.py:1362, in open_consolidated(store, metadata_key, mode, **kwargs)
   1361 # setup metadata store
-> 1362 meta_store = ConsolidatedStoreClass(store, metadata_key=metadata_key)
   1364 # pass through

File ~/miniconda3/envs/temp/lib/python3.12/site-packages/zarr/storage.py:3045, in ConsolidatedMetadataStore.__init__(self, store, metadata_key)
   3044 # retrieve consolidated metadata
-> 3045 meta = json_loads(self.store[metadata_key])
   3047 # check format of consolidated metadata

File ~/miniconda3/envs/temp/lib/python3.12/site-packages/zarr/storage.py:1120, in DirectoryStore.__getitem__(self, key)
   1119 else:
-> 1120     raise KeyError(key)
...
    137     return open_group(_store, mode=mode, **kwargs)
    138 else:
--> 139     raise PathNotFoundError(path)

PathNotFoundError: nothing found at path ''

[rcpeene]

Is there an estimated time for this patch to be released?

[rly]

We should have a patch released by the end of the week.