Question on using obstore for gcs-store read and write

[ofk123]

Hi, I was notified about obstore in one of the Pangeo weekly meetings, and eager to try it out.

After learning the API better, I did some very unofficial testing here, loading a 7GB zarr-v2-file, from a GCS-bucket in US-central region, using a S3 JupyterLab-deployment in Europe (Pangeo@EOSC).

The datasets I load in the example looks like this:

ds.info
[Out]  <bound method Dataset.info of <xarray.Dataset> Size: 7GB
        Dimensions:    (latitude: 720, longitude: 1440, pressure: 1, time: 216)
        Coordinates:
          * latitude   (latitude) float64 6kB -90.0 -89.75 -89.5 ... 89.25 89.5 89.75
          * longitude  (longitude) float64 12kB -180.0 -179.8 -179.5 ... 179.5 179.8
          * pressure   (pressure) float64 8B 1.1e+03
          * time       (time) datetime64[ns] 2kB 2004-01-16T12:00:00 ... 2021-12-16T1...
        Data variables:
            u          (latitude, longitude, time, pressure) float64 2GB dask.array<chunksize=(720, 1440, 24, 1), meta=np.ndarray>
            umse       (latitude, longitude, time, pressure) float64 2GB dask.array<chunksize=(720, 1440, 24, 1), meta=np.ndarray>
            v          (latitude, longitude, time, pressure) float64 2GB dask.array<chunksize=(720, 1440, 24, 1), meta=np.ndarray>
            vmse       (latitude, longitude, time, pressure) float64 2GB dask.array<chunksize=(720, 1440, 24, 1), meta=np.ndarray>
        Attributes:
            created_date:     2025-02-09 15:52:46.498249

• Would love to get general feedback on it so I can learn why the timing is so different?

[kylebarron]

Would you be able to test again using the obstore on latest main? If you install Rust then you should be able to just run

pip install git+https://github.com/developmentseed/obstore.git#subdirectory=obstore

There was some development in the underlying Rust library object_store that should've improved performance on GCS specifically.

[ofk123]

Hi @kylebarron , thanks for the suggestion!

I installed rust on the deployment (quay.io/pangeo/pangeo-notebook:2025-07-09

wget https://sh.rustup.rs -O rustup.sh
chmod +x rustup.sh
./rustup.sh

rustc --version
rustc 1.88.0 (6b00bc388 2025-06-23)

• Correct?

Then run
pip install git+https://github.com/developmentseed/obstore.git#subdirectory=obstore
But it seems it is already satisfied with

Collecting git+https://github.com/developmentseed/obstore.git#subdirectory=obstore
  Cloning https://github.com/developmentseed/obstore.git to /tmp/pip-req-build-jxym3lnt
  Running command git clone --filter=blob:none --quiet https://github.com/developmentseed/obstore.git /tmp/pip-req-build-jxym3lnt
  Resolved https://github.com/developmentseed/obstore.git to commit 583c48177bfc070496863f30c86b8cc809954aab
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: typing-extensions in /srv/conda/envs/notebook/lib/python3.12/site-packages (from obstore==0.7.0) (4.14.1)

The results look very similar to the version from the original test.

[kylebarron]

I think you may have to force install

[ofk123]

@kylebarron thanks.

Though, now I ran

pip install --force-reinstall git+https://github.com/developmentseed/obstore.git#subdirectory=obstore

which ended with the ERROR: Failed building wheel for obstore below..

Collecting git+https://github.com/developmentseed/obstore.git#subdirectory=obstore
  Cloning https://github.com/developmentseed/obstore.git to /tmp/pip-req-build-gyzntqpr
  Running command git clone --filter=blob:none --quiet https://github.com/developmentseed/obstore.git /tmp/pip-req-build-gyzntqpr
  Resolved https://github.com/developmentseed/obstore.git to commit 583c48177bfc070496863f30c86b8cc809954aab
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting typing-extensions (from obstore==0.7.0)
  Using cached typing_extensions-4.14.1-py3-none-any.whl.metadata (3.0 kB)
Using cached typing_extensions-4.14.1-py3-none-any.whl (43 kB)
Building wheels for collected packages: obstore
  Building wheel for obstore (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for obstore (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [38 lines of output]
      Running `maturin pep517 build-wheel -i /srv/conda/envs/notebook/bin/python3.12 --compatibility off`
      warning: Patch `object_store v0.12.2 (https://github.com/apache/arrow-rs-object-store?rev=f422dce1528ee2a089d8061af639c3f2a9cd43af#f422dce1)` was not used in the crate graph.
      Check that the patched package version and available features are compatible
      with the dependency requirements. If the patch has a different version from
      what is locked in the Cargo.lock file, run `cargo update` to use the new
      version. This may also occur with an optional dependency that is not enabled.
      🍹 Building a mixed python/rust project
      🔗 Found pyo3 bindings
      🐍 Found CPython 3.12 at /srv/conda/envs/notebook/bin/python3.12
      📡 Using build options features from pyproject.toml
      warning: Patch `object_store v0.12.2 (https://github.com/apache/arrow-rs-object-store?rev=f422dce1528ee2a089d8061af639c3f2a9cd43af#f422dce1)` was not used in the crate graph.
      Check that the patched package version and available features are compatible
      with the dependency requirements. If the patch has a different version from
      what is locked in the Cargo.lock file, run `cargo update` to use the new
      version. This may also occur with an optional dependency that is not enabled.
         Compiling proc-macro2 v1.0.95
         Compiling unicode-ident v1.0.18
         Compiling serde v1.0.219
         Compiling libc v0.2.174
         Compiling autocfg v1.5.0
         Compiling libm v0.2.15
         Compiling cfg-if v1.0.1
         Compiling bytes v1.10.1
         Compiling version_check v0.9.5
         Compiling once_cell v1.21.3
      error: linker `cc` not found
        |
        = note: No such file or directory (os error 2)
      
      error: could not compile `serde` (build script) due to 1 previous error
      warning: build failed, waiting for other jobs to finish...
      error: could not compile `libm` (build script) due to 1 previous error
      error: could not compile `proc-macro2` (build script) due to 1 previous error
      error: could not compile `libc` (build script) due to 1 previous error
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.12-64bit" PYO3_PYTHON="/srv/conda/envs/notebook/bin/python3.12" PYTHON_SYS_EXECUTABLE="/srv/conda/envs/notebook/bin/python3.12" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/pip-req-build-gyzntqpr/obstore/Cargo.toml" "--release" "--lib" "--" "-C" "strip=symbols"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/srv/conda/envs/notebook/bin/python3.12', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for obstore
Failed to build obstore
ERROR: Failed to build installable wheels for some pyproject.toml based projects (obstore)

[kylebarron]

Hmm perhaps you should use the develop instructions here. https://github.com/developmentseed/obstore/blob/main/DEVELOP.md

Regardless I plan to publish a new release soon

[kylebarron]

I published 0.7.1: https://github.com/developmentseed/obstore/releases/tag/py-v0.7.1

That should be the latest state of object_store

[ofk123]

Thanks.
I installed rust, cargo, and compilers, onto pangeo-notebook:2025-07-09, retried the timing.

The results look very similar to before.