ObjectStore to FsspecStore

[chitralverma]

Can a capability be added to convert an ObjectStore to an FsspecStore

Use: This can allow seamless usage in a code that involves fsspec only parts and non fsspec parts. For example, in my project, there are cases where I have to interact with GCS and I use the async list, list_with_delimiter etc. focusing on performance, but in other parts of the same code base i use with duckdb/ pyarrow to read data as well from same GCS as duckdb/ pyarrow both talk fsspec. So right now I create 2 separate handles and maintain them but an API like below can be very useful.

wdyt?

from obstore.store import ObjectStore
from obstore.fsspec import FsspecStore

# Option A
some_store: ObjectStore = ...
fs: FsspecStore = some_store.to_fsspec()

# Option B
some_store: ObjectStore = ...
fs: FsspecStore = FsspecStore.from_store(some_store)

# Option C (change the constructor of FsspecStore completely)
some_store: ObjectStore = ...
fs: FsspecStore = FsspecStore(some_store)

[kylebarron]

That's how we originally implemented FsspecStore, but it wasn't suitable for all cases because fsspec provides paths including the bucket, i.e. s3://bucket/path. Whereas ObjectStore is mounted on a single bucket. This means that FsspecStore needs to be able to create new ObjectStore instances depending on what bucket the user wants to access at runtime.

You can copy parameters from an object store instance to the fsspec store constructor:

from obstore.fsspec import FsspecStore
from obstore.store import S3Store

store = S3Store("sentinel-cogs", region="us-west-2", skip_signature=True)
prefix = "sentinel-s2-l2a-cogs/20/C/MQ/2025/2/S2B_20CMQ_20250208_0_L2A"
store.list_with_delimiter(prefix)
info1 = store.head(prefix + "/thumbnail.jpg")

args, kwargs = store.__getnewargs_ex__()
fsspec_store = FsspecStore("s3", *args, **kwargs)
info2 = fsspec_store.info(f"s3://sentinel-cogs/{prefix}/thumbnail.jpg")

assert info1["size"] == info2["size"]
assert info1["last_modified"] == info2["last_modified"]

The __getnewargs_ex__ implements pickle support, which returns the args, kwargs to construct a new object via __init__. Alternatively, to be clearer, you can do

kwargs = {
    "prefix": store.prefix,
    "config": store.config,
    "client_options": store.client_options,
    "retry_config": store.retry_config,
    "credential_provider": store.credential_provider,
}
fsspec_store = FsspecStore("s3", **kwargs)

Perhaps we could create a new constructor for FsspecStore that reads from this

[chitralverma]

i guess this also the reason why prefix is not set while creating fsspecstore instances directly

Alternatively, to be clearer, you can do

is there a way to get "s3" as well from a generic ObjectStore

Perhaps we could create a new constructor for FsspecStore that reads from this

that'll be great, thanks!

[kylebarron]

i guess this also the reason why prefix is not set while creating fsspecstore instances directly

I don't think fsspec stores have a meaning of a prefix given that fsspec always requires full paths.

is there a way to get "s3" as well from a generic ObjectStore

Yes and no. ObjectStore is intended to not leak the internal implementation details. In effect it should be anything that implements obspec methods. In practice, you can check isinstance(store, S3Store) though.