How to open Icechunk GCS store in Dask coiled

[nishadhka]

The GCS store, passing the dataset into the dask worker ends up in failed to serialize LLGExpr.

import icechunk
import xarray as xr

def tw_test_worker_icechunk_connection(dataset, test_name="basic_connection"):
    """Test that worker can initialize Icechunk connection independently"""
     data_vars = list(dataset.data_vars)
     shape = dict(dataset.sizes)
     # Test small computation
     spi_var = None
     for var in dataset.data_vars:
            if 'spi' in var.lower() or 'spc' in var.lower():
                spi_var = var
                break
        
        sample = dataset[spi_var].isel(time=0, lat=slice(0, 10), lon=slice(0, 10))
        return sample
        
storage = icechunk.gcs_storage(
            bucket=BUCKET_NAME,
            prefix=repo_prefix,
            service_account_file=SERVICE_ACCOUNT_FILE  # Use uploaded file path
        )

repo = icechunk.Repository.open(storage)
session = repo.readonly_session("main")  
#ds = xr.open_zarr(session.store, group="spi9_data", zarr_format=3, consolidated=False)
ds = xr.open_dataset(session.store, group="spi9_data",engine="zarr", zarr_format=3, consolidated=False)

future30 = client.submit(
    tw_test_worker_icechunk_connection,
    ds, "basic_test30"
)

ends up n error

2025-07-23 00:43:44,539 - distributed.protocol.pickle - ERROR - Failed to serialize LLGExpr(dsk={'tw_test_worker_icechunk_connection-2adefb2e1690a5d07088983197a16a6c': <Task 'tw_test_worker_icechunk_connection-2adefb2e1690a5d07088983197a16a6c' tw_test_worker_icechunk_connection(...)>}).
Traceback (most recent call last):
  File "[/opt/coiled/env/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 60](https://cluster-bbdxu.dask.host/opt/coiled/env/lib/python3.11/site-packages/distributed/protocol/pickle.py#line=59), in dumps
    result = pickle.dumps(x, **dump_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[/opt/coiled/env/lib/python3.11/site-packages/icechunk/store.py", line 81](https://cluster-bbdxu.dask.host/opt/coiled/env/lib/python3.11/site-packages/icechunk/store.py#line=80), in __getstate__
    d["_store"] = self._store.as_bytes()
                  ^^^^^^^^^^^^^^^^^^^^^^
icechunk.IcechunkError:   x serialization error
  | 
  | context:
  |    0: icechunk::store::as_bytes
  |              at icechunk[/src/store.rs:169](https://cluster-bbdxu.dask.host/src/store.rs#line=168)
  | 
  |-> serialization error
  `-> cannot serialize tagged newtype variant GcsStaticCredentials::ServiceAccount containing a string


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "[/opt/coiled/env/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 65](https://cluster-bbdxu.dask.host/opt/coiled/env/lib/python3.11/site-packages/distributed/protocol/pickle.py#line=64), in dumps
    pickler.dump(x)
  File "[/opt/coiled/env/lib/python3.11/site-packages/icechunk/store.py", line 81](https://cluster-bbdxu.dask.host/opt/coiled/env/lib/python3.11/site-packages/icechunk/store.py#line=80), in __getstate__
    d["_store"] = self._store.as_bytes()
                  ^^^^^^^^^^^^^^^^^^^^^^
icechunk.IcechunkError:   x serialization error
  | 
  | context:
  |    0: icechunk::store::as_bytes
  |              at icechunk[/src/store.rs:169](https://cluster-bbdxu.dask.host/src/store.rs#line=168)
  | 
  |-> serialization error
  `-> cannot serialize tagged newtype variant GcsStaticCredentials::ServiceAccount containing a string


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "[/opt/coiled/env/lib/python3.11/site-packages/distributed/protocol/pickle.py", line 77](https://cluster-bbdxu.dask.host/opt/coiled/env/lib/python3.11/site-packages/distributed/protocol/pickle.py#line=76), in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[/opt/coiled/env/lib/python3.11/site-packages/cloudpickle/cloudpickle.py", line 1537](https://cluster-bbdxu.dask.host/opt/coiled/env/lib/python3.11/site-packages/cloudpickle/cloudpickle.py#line=1536), in dumps
    cp.dump(obj)
  File "[/opt/coiled/env/lib/python3.11/site-packages/cloudpickle/cloudpickle.py", line 1303](https://cluster-bbdxu.dask.host/opt/coiled/env/lib/python3.11/site-packages/cloudpickle/cloudpickle.py#line=1302), in dump
    return super().dump(obj)
           ^^^^^^^^^^^^^^^^^
  File "[/opt/coiled/env/lib/python3.11/site-packages/icechunk/store.py", line 81](https://cluster-bbdxu.dask.host/opt/coiled/env/lib/python3.11/site-packages/icechunk/store.py#line=80), in __getstate__
    d["_store"] = self._store.as_bytes()
                  ^^^^^^^^^^^^^^^^^^^^^^
icechunk.IcechunkError:   x serialization error
  | 
  | context:
  |    0: icechunk::store::as_bytes
  |              at icechunk[/src/store.rs:169](https://cluster-bbdxu.dask.host/src/store.rs#line=168)
  | 
  |-> serialization error
  `-> cannot serialize tagged newtype variant GcsStaticCredentials::ServiceAccount containing a string

---------------------------------------------------------------------------
IcechunkError                             Traceback (most recent call last)
File [/opt/coiled/env/lib/python3.11/site-packages/distributed/protocol/pickle.py:60](https://cluster-bbdxu.dask.host/opt/coiled/env/lib/python3.11/site-packages/distributed/protocol/pickle.py#line=59), in dumps(x, buffer_callback, protocol)
     59 try:
---> 60     result = pickle.dumps(x, **dump_kwargs)
     61 except Exception:

File [/opt/coiled/env/lib/python3.11/site-packages/icechunk/store.py:81](https://cluster-bbdxu.dask.host/opt/coiled/env/lib/python3.11/site-packages/icechunk/store.py#line=80), in IcechunkStore.__getstate__(self)
     80 # we serialize the Rust store as bytes
---> 81 d["_store"] = self._store.as_bytes()
     82 d["_for_fork"] = self._for_fork

If the store opened inside the worker ends up in Could not serialize object of type _HLGExprSequence

# This fails because the storage object can't be serialized
def test_icechunk_access_on_worker():
    storage = icechunk.gcs_storage(
        bucket=BUCKET_NAME,
        prefix=repo_prefix,
        service_account_file=worker_creds_path
    )
    repo = icechunk.Repository.open(storage)
    dataset = xr.open_zarr(session.store, group=group_name)
    # Any operation on dataset will fail when serialized
    return float(dataset[var].isel(time=0, lat=0, lon=0).values)

# Submitting this function fails
future = client.submit(test_icechunk_access_on_worker)

ends up in error

Icechunk failed on worker:
   Error: Could not serialize object of type _HLGExprSequence
   Type: TypeError
icechunk.IcechunkError: x serialization error
  |-> serialization error
  |
  `-> cannot serialize tagged newtype variant GcsStaticCredentials::ServiceAccount containing a string
  |              at icechunk/src/store.rs:169
  | context:
  |    0: icechunk::store::as_bytes
File "/opt/coiled/env/lib/python3.11/site-packages/icechunk/store.py", line 81, in __getstate__
  d["_store"] = self._store.as_bytes()
  ^^^^^^^^^^^^^^^^^^^^^^
icechunk.IcechunkError: x serialization error
  |-> serialization error
  |
  `-> cannot serialize tagged newtype variant GcsStaticCredentials::ServiceAccount containing a string

Versions
dask version: 2025.5.0
numpy version: 2.2.6
gcsfs version: 2025.5.1
zarr version: 3.1.0
icechunk version: 1.0.0

[mpiannucci]

Thanks for reporting! Can you give an example of the value you provided for the service_account_file? Was it a Linux file path?

[nishadhka]

It is GCP service account file in json format JSON Web Token (JWT). Like service_account_file='/home/user/service_account.json'
https://cloud.google.com/iam/docs/service-account-creds, the path to the json file

[mpiannucci]

I have confirmed this is a bug in icechunk, I am working on a fix. Sorry about that!

[mpiannucci]

Hi @nishadhka, we just released a fix for this with icehcunk version 1.0.2, can you give that a try and report back?

[nishadhka]

Thank you, this is solved @mpiannucci, the .compute() was error earlier, now working perfect.
Checked on creating icechunk inside the dask worker and passing the icechunk dataset to worker.
In version 1.0.0, Interestingly the credentials gets passed upto detailing the dataset, get metadata, such as variables and shapes, but error happens only with .compute(), numpy.array.load, likely with data pickle, will check with the changes to understand.