Support external pyo3-object_store in obstore

[aseyboldt]

This is more a feature request than an issue.

I work on a rust library (nutpie) that can write its results to a zarr store with zarrs.
The library is usually used from python, so pyo3-object_store is perfect to let me configure the store in the python part of the library, pass it to rust and let the rust part write the results to the store.

However, as I found out after writing the whole new zarr backend, I don't think I can then use the resulting store from python with zarr. zarr/xarray does support obstore, but strictly requires that the store was created in obstore, and does not accept a store created from a different rust library with pyo3-object_store.

For file based storage, I can simply recreate a separate store that points to the written files, but I think for other object stores that will be quite inefficient, and for the memory store (which I think has to be the default in my application) it does not work at all. I think I can also not safely access the store while it is still being written to, so I can't give users the ability to see intermediate results.

I'm not sure how difficult it would be to support a use case like this?
Maybe there could be a function like the existing register_store_module, that exports the functions of obstore itself?

[kylebarron]

We can discuss whether to allow external object stores as constructors into obstore, but the underlying implementation uses pickling and so is in-effect public.

You can just call

store = S3Store(...)
args, kwargs = store.__getnewargs_ex__()
new_store = S3Store(*args, **kwargs)

And that's all the "import from external library" does.

obstore/pyo3-object_store/src/store.rs

Lines 97 to 107 in fbaeb7f

	if cls_name == PyAzureStore::NAME {
	let (args, kwargs): (Bound<PyTuple>, Bound<PyDict>) = ob
	.call_method0(intern!(py, "__getnewargs_ex__"))?
	.extract()?;
	let store = PyAzureStore::type_object(py)
	.call(args, Some(&kwargs))?
	.downcast::<PyAzureStore>()?
	.get()
	.clone();
	return Ok(Self(store.into_inner()));
	}

In theory we were going to explore updating the Zarr code to accept any obstore-like class implementing https://github.com/developmentseed/obspec, but nothing's moved on that.

Maybe there could be a function like the existing register_store_module, that exports the functions of obstore itself?

I'm not sure I'm ready to export the Python API as a public API to other Rust libraries. That would make it harder to evolve on the Python API.

[aseyboldt]

In theory we were going to explore updating the Zarr code to accept any obstore-like class implementing https://github.com/developmentseed/obspec, but nothing's moved on that.

I think that's pretty much what I had in mind (only better): A rust function that exports a python module that is compatible with obstore.

Do you know if it is safe to read a zarr file system archive with obstore while it is being written to from rust in an external store, assuming nothing changes metadata anymore? If that's the case, I could at least use this as-is with stores other than MemoryStore.

[kylebarron]

Do you know if it is safe to read a zarr file system archive with obstore while it is being written to from rust in an external store, assuming nothing changes metadata anymore? If that's the case, I could at least use this as-is with stores other than MemoryStore.

This is unrelated to obstore, but no, since Zarr is a directory of files, it doesn't provide ACID guarantees so you can't read and write to the same Zarr dataset at the same time. For that you'd want to look at something like Icechunk