Cannot produce multiscale sharded volumes

[LucaMarconato]

I have a small single-scale, unsharded volume that looks like this:

.
├── 100_100_100
├── info
└── provenance

where this is the info file:

{                                                                                                                                       
  "data_type": "uint16",
  "num_channels": 1,
  "scales": [
    {
      "chunk_sizes": [
        [
          512,
          512,
          194
        ]
      ],
      "encoding": "raw",
      "key": "100_100_100",
      "resolution": [
        100,
        100,
        100
      ],
      "size": [
        5440,
        4792,
        194
      ],
      "voxel_offset": [
        0,
        0,
        0
      ]
    }
  ],
  "type": "segmentation"
}

I can successfully use igneous to create a multi-scale unsharded volume with

igneous image downsample . --queue my-queue && igneous execute -x my-queue

I can also create a single-scale, sharded volume using

igneous image xfer . new_sharded_volume --sharded --queue my-queue && igneous execute -x my-queue

But if I try computing a multiscale, sharded volume, with

igneous image downsample . --sharded --queue my-queue && igneous execute -x my-queue           

I get the error ValueError: buffer source array is read-only.

Full traceback.

(tissue-map-tools) macbook@MacBook-Pro-64 invasive_precomputed % igneous image downsample . --sharded      
Volume Bounds:  Bbox([0, 0, 0],[5440, 4792, 194], dtype=np.int32, unit='vx')
Selected ROI:   Bbox([0, 0, 0],[5440, 4792, 194], dtype=np.int32, unit='vx')
Tasks:   0%|                                                                                                      | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/bin/igneous", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/igneous_cli/cli.py", line 316, in downsample
    enqueue_tasks(ctx, queue, tasks)
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/igneous_cli/cli.py", line 74, in enqueue_tasks
    tq.insert_all(tasks)
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/taskqueue/taskqueue.py", line 458, in insert_all
    self.execute(self.progress)
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/taskqueue/taskqueue.py", line 480, in execute
    _task_execute(self.queue.pop(0))
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/taskqueue/taskqueue.py", line 508, in _task_execute
    task.execute(*args, **kwargs)
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/taskqueue/queueablefns.py", line 78, in execute
    self(*args, **kwargs)
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/taskqueue/queueablefns.py", line 87, in __call__
    return self.tofunc()()
           ^^^^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/igneous/tasks/image/image.py", line 739, in ImageShardDownsampleTask
    img, mapping = src_vol.download(
                   ^^^^^^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/cloudvolume/frontends/precomputed.py", line 818, in download
    tup = self.image.download(
          ^^^^^^^^^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/cloudvolume/datasource/precomputed/image/__init__.py", line 220, in download
    return rx.download(
           ^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 314, in download
    download_chunks_threaded(
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 667, in download_chunks_threaded
    schedule_jobs(
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/cloudvolume/scheduler.py", line 145, in schedule_jobs
    return schedule_single_threaded_jobs(fns, progress, total)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/cloudvolume/scheduler.py", line 112, in schedule_single_threaded_jobs
    res = fn()
          ^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 626, in process
    fn(labels, bbox)
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/cloudvolume/datasource/precomputed/image/rx.py", line 307, in process_renumber
    fastremap.remap(img3d, remap, in_place=True)
  File "src/fastremap.pyx", line 693, in fastremap.remap
  File "src/fastremap.pyx", line 696, in fastremap._remap
ValueError: buffer source array is read-only

Any advice on how to proceed?

[william-silversmith]

I believe we ran into this issue a while back. Try upgrading cloud-volume and fastremap and see if that fixes it.

[LucaMarconato]

Amazing thanks! Using

cloud-volume==12.6.2
fastremap==1.17.7

made the problem disappear.

(I am now encountering a different error from morton_codes(), but before reporting it I'll try to see if I have done something wrong with the data while attempting to investigate the above).

[LucaMarconato]

I retried with clean workflow and the error with morton_codes() keeps occurring. I will actually re-open and discuss this second problem in this issue since the title of the issue applies also to the new one.

I now get the following error, using the same code (but the new package versions). Do you have an insight on how to address it?

Volume Bounds:  Bbox([0, 0, 0],[5440, 4792, 194], dtype=np.int32, unit='vx')
Selected ROI:   Bbox([0, 0, 0],[5440, 4792, 194], dtype=np.int32, unit='vx')
Tasks:   0%|          | 0/1 [00:20<?, ?it/s]
multiprocess.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/pathos/helpers/mp_helper.py", line 15, in <lambda>
    func = lambda args: f(*args)
                        ^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/taskqueue/taskqueue.py", line 508, in _task_execute
    task.execute(*args, **kwargs)
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/taskqueue/queueablefns.py", line 78, in execute
    self(*args, **kwargs)
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/taskqueue/queueablefns.py", line 87, in __call__
    return self.tofunc()()
           ^^^^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/igneous/tasks/image/image.py", line 809, in ImageShardDownsampleTask
    chunk_dict = src_vol.image.make_shard_chunks(shard_cutout, shard_bbox, mip + i + 1)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/cloudvolume/datasource/precomputed/image/__init__.py", line 704, in make_shard_chunks
    gpts, morton_codes = self.morton_codes(bbox, mip=mip, spec=spec)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/cloudvolume/datasource/precomputed/image/__init__.py", line 650, in morton_codes
    raise exceptions.AlignmentError(
cloudvolume.exceptions.AlignmentError: Unable to create shard from a non-chunk aligned bounding box. Requested: Bbox([0, 0, 0],[2816, 2560, 194], dtype=np.int32, unit='vx'), Aligned: Bbox([0, 0, 0],[3072, 2560, 194], dtype=np.int32, unit='vx')
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/examples/invasive/ome_tiff_to_sharded_meshes.py", line 122, in <module>
    from_precomputed_raster_to_multiscale_precomputed_raster(
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/src/tissue_map_tools/igneous_converters.py", line 53, in from_precomputed_raster_to_multiscale_precomputed_raster
    task_queue.execute()
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/taskqueue/taskqueue.py", line 484, in execute
    for _ in executor.imap(_task_execute, self.queue):
  File "/Users/macbook/embl/projects/basel/tissue-map-tools/.venv/lib/python3.11/site-packages/multiprocess/pool.py", line 873, in next
    raise value
cloudvolume.exceptions.AlignmentError: Unable to create shard from a non-chunk aligned bounding box. Requested: Bbox([0, 0, 0],[2816, 2560, 194], dtype=np.int32, unit='vx'), Aligned: Bbox([0, 0, 0],[3072, 2560, 194], dtype=np.int32, unit='vx')

Process finished with exit code 1

[william-silversmith]

I'm not sure, I recently revamped the downsampling code to support multi-mip generation, but it may have introduced new errors too. Try downgrading Igneous to 4.32.0 and see if that fixes it.

[LucaMarconato]

Using 4.32.0 indeed fixes the problem, and I am now able to generate the scales (with multiple calls of the igneous APIs). Thanks!

For reference, this is the code I am now using:

for i in range(num_mips):
    cv = CloudVolume(data_path, mip=i)
    factor = get_downsampling_factor(cv.shape)
    if sharded:
        task = create_image_shard_downsample_tasks(
            cloudpath=data_path,
            factor=factor,
            mip=i,
        )
    else:
        task = create_downsampling_tasks(
            layer_path=data_path,
            mip=i,
            num_mips=1,
            factor=factor,
        )
    task_queue.insert(task)
    task_queue.execute()

[william-silversmith]

Ok, thanks! Another colleague reported a similar bug, I'll have to spend some time fixing it in the near future.