ITKWASM_GAUSSIAN and tensorstore failing for large datasets

[swelborn]

Hello. I'm getting two different exceptions when trying out the tensorstore saving, and wasm gaussian downsampling.

Gemini and I made a little reproducer here: https://github.com/swelborn/ngff_wasm_tensorcore_reproducer

This is the output for both test cases (separated by issue).

Wasm issue:

2025-04-29 19:14:51,047 - ZarrReproducer - WARNING - Removing base output directory: zarr_reproducer_output
2025-04-29 19:14:51,047 - ZarrReproducer - INFO - Creating random small data with shape=(64, 1024, 1024), dtype=<class 'numpy.float32'>...
2025-04-29 19:14:51,321 - ZarrReproducer - INFO - Input data created (0.250 GB) in 0.27 seconds.
2025-04-29 19:14:51,321 - ZarrReproducer - INFO - 


--- Running Test Case: itkwasm_gaussian_multiscale_failure ---
2025-04-29 19:14:51,321 - ZarrReproducer - INFO -   Input Shape: (64, 1024, 1024)
2025-04-29 19:14:51,321 - ZarrReproducer - INFO -   Multiscale Method: ITKWASM_GAUSSIAN
2025-04-29 19:14:51,321 - ZarrReproducer - INFO -   Creating NGFF image object...
2025-04-29 19:14:51,448 - ZarrReproducer - INFO -   Creating multiscales (expecting failure)...
dataSize cannot be zero or negative.
2025-04-29 19:14:51,600 - ZarrReproducer - INFO -   EXPECTED FAILURE: Test case 'itkwasm_gaussian_multiscale_failure' failed as expected after 0.28 seconds.
2025-04-29 19:14:51,600 - ZarrReproducer - INFO -   Caught expected WasmTrap: error while executing at wasm backtrace:
    0: 0x16abc2 - <unknown>!<wasm function 1905>
    1: 0x2d70fa - <unknown>!<wasm function 3827>
    2: 0x2d8177 - <unknown>!<wasm function 3829>

Caused by:
    wasm trap: wasm `unreachable` instruction executed
2025-04-29 19:14:51,600 - ZarrReproducer - ERROR -   Error details: error while executing at wasm backtrace:
    0: 0x16abc2 - <unknown>!<wasm function 1905>
    1: 0x2d70fa - <unknown>!<wasm function 3827>
    2: 0x2d8177 - <unknown>!<wasm function 3829>

Caused by:
    wasm trap: wasm `unreachable` instruction executed
Traceback (most recent call last):
  File "main.py", line 84, in test_itkwasm_gaussian_failure
    multiscales = to_multiscales(
                  ^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/ngff_zarr/to_multiscales.py", line 326, in to_multiscales
    images = _downsample_itkwasm(
             ^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/ngff_zarr/methods/_itkwasm.py", line 130, in _downsample_itkwasm
    block_output = downsample_bin_shrink(
                   ^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/itkwasm_downsample/downsample_bin_shrink.py", line 31, in downsample_bin_shrink
    output = func(input, shrink_factors=shrink_factors, information_only=information_only)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/itkwasm_downsample_cucim/downsample_bin_shrink.py", line 33, in downsample_bin_shrink
    result = downsample_bin_shrink_wasi(wasi_input, shrink_factors, information_only=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/itkwasm_downsample_wasi/downsample_bin_shrink.py", line 70, in downsample_bin_shrink
    outputs = _pipeline.run(args, pipeline_outputs, pipeline_inputs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/itkwasm/pipeline.py", line 418, in run
    return_code = ri.delayed_start()
                  ^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/itkwasm/pipeline.py", line 152, in delayed_start
    return_code = func(self._store)
                  ^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/wasmtime/_func.py", line 93, in __call__
    with enter_wasm(store) as trap:
  File "cpython-3.11.12-linux-x86_64-gnu/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File ".venv/lib/python3.11/site-packages/wasmtime/_func.py", line 265, in enter_wasm
    raise trap_obj
wasmtime._trap.Trap: error while executing at wasm backtrace:
    0: 0x16abc2 - <unknown>!<wasm function 1905>
    1: 0x2d70fa - <unknown>!<wasm function 3827>
    2: 0x2d8177 - <unknown>!<wasm function 3829>

Caused by:
    wasm trap: wasm `unreachable` instruction executed

Tensorstore issue:

2025-04-29 19:14:51,601 - ZarrReproducer - INFO - Creating random large data with shape=(1024, 2048, 2048), dtype=<class 'numpy.float32'>...
2025-04-29 19:15:09,152 - ZarrReproducer - INFO - Input data created (16.000 GB) in 17.55 seconds.
2025-04-29 19:15:09,153 - ZarrReproducer - INFO - 
--- Running Test Case: tensorstore_already_exists_write_failure ---
2025-04-29 19:15:09,153 - ZarrReproducer - INFO -   Input Shape: (1024, 2048, 2048)
2025-04-29 19:15:09,153 - ZarrReproducer - INFO -   Multiscale Method: DASK_IMAGE_GAUSSIAN (using a working method)
2025-04-29 19:15:09,153 - ZarrReproducer - INFO -   Use TensorStore: True
2025-04-29 19:15:09,153 - ZarrReproducer - INFO -   Output Path: zarr_reproducer_output/tensorstore_already_exists_write_failure/651c905f.zarr
2025-04-29 19:15:09,153 - ZarrReproducer - INFO -   Creating NGFF image object...
2025-04-29 19:15:17,094 - ZarrReproducer - INFO -   Creating multiscales...
2025-04-29 19:15:17,533 - ZarrReproducer - INFO -   Multiscales created in 0.44 seconds.
2025-04-29 19:15:17,534 - ZarrReproducer - INFO -   Writing NGFF Zarr with TensorStore (expecting failure)...
2025-04-29 19:15:17,552 - ZarrReproducer - INFO -   EXPECTED FAILURE: Test case 'tensorstore_already_exists_write_failure' failed as expected after 8.40 seconds.
2025-04-29 19:15:17,552 - ZarrReproducer - INFO -   Caught expected ValueError (ALREADY_EXISTS): ALREADY_EXISTS: Error opening "zarr" driver: Error writing local file "zarr_reproducer_output/tensorstore_already_exists_write_failure/651c905f.zarr/scale0/image/.zarray" [tensorstore_spec='{\"context\":{\"cache_pool\":{},\"data_copy_concurrency\":{},\"file_io_concurrency\":{},\"file_io_locking\":{},\"file_io_memmap\":false,\"file_io_sync\":true},\"create\":true,\"driver\":\"zarr\",\"dtype\":\"float32\",\"kvstore\":{\"driver\":\"file\",\"path\":\"zarr_reproducer_output/tensorstore_already_exists_write_failure/651c905f.zarr/scale0/image/\"},\"metadata\":{\"chunks\":[128,128,128],\"dimension_separator\":\"/\",\"dtype\":\"<f4\",\"shape\":[128,128,128]},\"transform\":{\"input_exclusive_max\":[[128],[128],[128]],\"input_inclusive_min\":[0,0,0]}}'] [source locations='tensorstore/internal/cache/kvs_backed_cache.h:220\ntensorstore/driver/driver.cc:120']
2025-04-29 19:15:17,552 - ZarrReproducer - ERROR -   Error details: ALREADY_EXISTS: Error opening "zarr" driver: Error writing local file "zarr_reproducer_output/tensorstore_already_exists_write_failure/651c905f.zarr/scale0/image/.zarray" [tensorstore_spec='{\"context\":{\"cache_pool\":{},\"data_copy_concurrency\":{},\"file_io_concurrency\":{},\"file_io_locking\":{},\"file_io_memmap\":false,\"file_io_sync\":true},\"create\":true,\"driver\":\"zarr\",\"dtype\":\"float32\",\"kvstore\":{\"driver\":\"file\",\"path\":\"zarr_reproducer_output/tensorstore_already_exists_write_failure/651c905f.zarr/scale0/image/\"},\"metadata\":{\"chunks\":[128,128,128],\"dimension_separator\":\"/\",\"dtype\":\"<f4\",\"shape\":[128,128,128]},\"transform\":{\"input_exclusive_max\":[[128],[128],[128]],\"input_inclusive_min\":[0,0,0]}}'] [source locations='tensorstore/internal/cache/kvs_backed_cache.h:220\ntensorstore/driver/driver.cc:120']
Traceback (most recent call last):
  File "main.py", line 156, in test_tensorstore_already_exists_failure
    to_ngff_zarr(
  File ".venv/lib/python3.11/site-packages/ngff_zarr/to_ngff_zarr.py", line 488, in to_ngff_zarr
    _write_with_tensorstore(
  File ".venv/lib/python3.11/site-packages/ngff_zarr/to_ngff_zarr.py", line 155, in _write_with_tensorstore
    dataset = ts.open(spec, create=True, dtype=array.dtype).result()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: ALREADY_EXISTS: Error opening "zarr" driver: Error writing local file "zarr_reproducer_output/tensorstore_already_exists_write_failure/651c905f.zarr/scale0/image/.zarray" [tensorstore_spec='{\"context\":{\"cache_pool\":{},\"data_copy_concurrency\":{},\"file_io_concurrency\":{},\"file_io_locking\":{},\"file_io_memmap\":false,\"file_io_sync\":true},\"create\":true,\"driver\":\"zarr\",\"dtype\":\"float32\",\"kvstore\":{\"driver\":\"file\",\"path\":\"zarr_reproducer_output/tensorstore_already_exists_write_failure/651c905f.zarr/scale0/image/\"},\"metadata\":{\"chunks\":[128,128,128],\"dimension_separator\":\"/\",\"dtype\":\"<f4\",\"shape\":[128,128,128]},\"transform\":{\"input_exclusive_max\":[[128],[128],[128]],\"input_inclusive_min\":[0,0,0]}}'] [source locations='tensorstore/internal/cache/kvs_backed_cache.h:220\ntensorstore/driver/driver.cc:120']
2025-04-29 19:15:17,603 - ZarrReproducer - INFO - 
--- Reproducer Summary ---
2025-04-29 19:15:17,604 - ZarrReproducer - INFO - ITKWASM Gaussian Failure Test: EXPECTED FAILURE OCCURRED
2025-04-29 19:15:17,604 - ZarrReproducer - INFO - TensorStore ALREADY_EXISTS Test: EXPECTED FAILURE OCCURRED
2025-04-29 19:15:17,604 - ZarrReproducer - INFO - -------------------------

Edit/update:

Removing the cucim-related dependencies takes out the first exception. Is the cucim version still supported?

[thewtex]

@swelborn thanks for the report and reproducer 🥇

Removing the cucim-related dependencies takes out the first exception. Is the cucim version still supported?

We need to get CI for it.

[swelborn]

Gotcha, thanks :)

Do you think your refactor here may address the tensorstore issue? #151

[thewtex]

I don't necessarily expect it to address the issue, but help testing is appreciated. #151 is merged.

I will help with #149 , then circle back to this.

[swelborn]

hey @thewtex would you mind cutting a release for #151? would make testing a bit more straightforward on my end

[thewtex]

hey @swelborn yes, published in 0.13.1.

[thewtex]

@swelborn patch in #155

[swelborn]

ty @thewtex i will check it out

[swelborn]

seem like both of the issues are fixed.

[thewtex]

@swelborn thanks for checking!