Unreadable shards containing partial chunks

[malosellier]

Hello Icechunk community,

Many thanks to all the devs for this excellent work!
As a first appearance here, I'd like to ask a question about shard and chunk sizes as it seems that Icechunk does not handle "partial" shards.

As an example, the code below creates 3 array (lat/lon dimensions + variable), without Icechunk and the output is a readable xarray.Dataset.

output_path=f"/output/path/file1.zarr"
store = zarr.storage.LocalStore(root=output_path)

lats = zarr.create_array(name="lat",
                       store=store,
                       dimension_names=['lat'],
                       shape=(1001),
                       chunks=(10,),
                       shards=(100,),
                       dtype='float32',
                       overwrite=True)
lats[:] = np.linspace(35, 70, 1001)

lons = zarr.create_array(name="lon",
                       store=store,
                       dimension_names=['lon'],
                       shape=(1001),
                       chunks=(10,),
                       shards=(100,),
                       dtype='float32',
                       overwrite=True)
lons[:] = np.linspace(-10, 30, 1001)

var = zarr.create_array(name="variable_name",
                       store=store,
                       dimension_names=['lat', 'lon'],
                       shape=(1001, 1001),
                       chunks=(10, 10),
                       shards=(100, 100),
                       dtype='float32',
                       overwrite=True)
var[:] = np.random.randint(0, 100, size=(1001, 1001))

ds_zarr = xr.open_zarr(store=store, consolidated=False)
ds_zarr

Here is a very similar script using Icechunk to write the zarr file:

output_path=f"/output/path/file2.zarr"

storage = icechunk.local_filesystem_storage(output_path)
repo = icechunk.Repository.open_or_create(storage=storage)
session = repo.writable_session("main")
store = session.store

lats = zarr.create_array(name="lat",
                       store=store,
                       dimension_names=['lat'],
                       shape=(1001),
                       chunks=(10,),
                       shards=(100,),
                       dtype='float32',
                       overwrite=True)
lats[:] = np.linspace(35, 70, 1001)

lons = zarr.create_array(name="lon",
                       store=store,
                       dimension_names=['lon'],
                       shape=(1001),
                       chunks=(10,),
                       shards=(100,),
                       dtype='float32',
                       overwrite=True)
lons[:] = np.linspace(-10, 30, 1001)

var = zarr.create_array(name="variable_name",
                       store=store,
                       dimension_names=['lat', 'lon'],
                       shape=(1001, 1001),
                       chunks=(10, 10),
                       shards=(100, 100),
                       dtype='float32',
                       overwrite=True)
var[:] = np.random.randint(0, 100, size=(1001, 1001))
session.commit("Created array with icechunk")

session = repo.readonly_session("main")
store = session.store

ds_zarr = xr.open_zarr(store=store, consolidated=False)
ds_zarr

This last script gives the following error: ValueError: Stored and computed checksum do not match. Stored: b'\x00H\xa0\x05'. Computed: b'\x95;\xbf\xb6'. However, the exact same script using (1000, 1000) as shape instead of (1001, 1001) works fine.

Thus, I wonder what is happening there? There is probably a good reason for this not working and I'd understand it.
Thank you again,
Malo

Edit: in case this may impact:

zarr.__version__ = '3.0.8'
icechunk.__version__ = '0.2.15'
xarray.__version__ = '2025.4.0'

[jhamman]

Thanks @malosellier for opening this issue and the nice example. We'll take a look and get back to you.

[jhamman]

Tracking this in a bug report here: #1019

[malosellier]

Closing discussion as answered in #1019