Skip to content

[Feature]: Add tests for reading from S3 bucket with references #252

Open
@rly

Description

@rly

What would you like to see added to HDMF-ZARR?

See #238 and NeurodataWithoutBorders/nwb_benchmarks#77

Error:

For parameters: 's3://dandiarchive/zarr/18e75d22-f527-4051-a4c8-c7e0f1e7dad1/'
             . Using tshark at: /Applications/Wireshark.app/Contents/MacOS/tshark
             Traceback (most recent call last):
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv/benchmark.py", line 68, in <module>
                 main()
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv/benchmark.py", line 60, in main
                 commands[mode](args)
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv_runner/run.py", line 72, in _run
                 result = benchmark.do_run()
                          ^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv_runner/benchmarks/_base.py", line 661, in do_run
                 return self.run(*self._current_params)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv_runner/benchmarks/track.py", line 64, in run
                 return self.func(*param)
                        ^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv_runner/benchmarks/mark.py", line 105, in wrapper
                 return func(*args, **kwargs)
                        ^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv_runner/benchmarks/mark.py", line 164, in wrapper
                 return func(*args, **kwargs)
                        ^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/nwb_benchmarks/src/nwb_benchmarks/benchmarks/network_tracking_remote_file_reading.py", line 327, in track_network_activity_during_read
                 self.nwbfile, self.io = read_zarr_nwbfile(s3_url=s3_url, mode="r")
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/nwb_benchmarks/src/nwb_benchmarks/core/_streaming.py", line 264, in read_zarr_nwbfile
                 nwbfile = io.read()
                           ^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/hdmf/utils.py", line 668, in func_call
                 return func(args[0], **pargs)
                        ^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/hdmf/backends/io.py", line 56, in read
                 f_builder = self.read_builder()
                             ^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/hdmf/utils.py", line 668, in func_call
                 return func(args[0], **pargs)
                        ^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/hdmf-zarr/src/hdmf_zarr/backend.py", line 1467, in read_builder
                 f_builder = self.__read_group(self.__file, ROOT_NAME)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/hdmf-zarr/src/hdmf_zarr/backend.py", line 1545, in __read_group
                 sub_builder = self.__read_group(sub_group, sub_name)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/hdmf-zarr/src/hdmf_zarr/backend.py", line 1545, in __read_group
                 sub_builder = self.__read_group(sub_group, sub_name)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/hdmf-zarr/src/hdmf_zarr/backend.py", line 1554, in __read_group
                 self.__read_links(zarr_obj=zarr_obj, parent=ret)
               File "/Users/rly/Documents/NWB/hdmf-zarr/src/hdmf_zarr/backend.py", line 1573, in __read_links
                 target_name, target_zarr_obj = self.resolve_ref(link)
                                                ^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/hdmf-zarr/src/hdmf_zarr/backend.py", line 819, in resolve_ref
                 target_zarr_obj = self.__open_file_consolidated(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/Documents/NWB/hdmf-zarr/src/hdmf_zarr/backend.py", line 566, in __open_file_consolidated
                 return zarr.open_consolidated(
                        ^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/zarr/convenience.py", line 1345, in open_consolidated
                 store = normalize_store_arg(
                         ^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/zarr/storage.py", line 201, in normalize_store_arg
                 return normalize_store(store, storage_options, mode)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               File "/Users/rly/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/zarr/storage.py", line 173, in _normalize_store_arg_v2
                 raise ValueError("storage_options passed with non-fsspec path")
             ValueError: storage_options passed with non-fsspec path

Is your feature request related to a problem?

No response

What solution would you like?

Add tests for this

Do you have any interest in helping implement the feature?

Yes.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions