Skip to content

FsspecStore fails for local URL-style windows paths #554

@keen85

Description

@keen85

Hi,
I noticed the following bug when working with local Windows URL-style paths:

✅works fine

import fsspec

local_path = "file:///C:/dev/tmp"

fs_fsspec = fsspec.filesystem("file")
fs_fsspec.exists(local_path)

❌ fails

from obstore.fsspec import FsspecStore

local_path = "file:///C:/dev/tmp"

fs_fsspecstore = FsspecStore("file")
fs_fsspecstore.exists(local_path)
---------------------------------------------------------------------------
GenericError                              Traceback (most recent call last)
Cell In[18], line 10
      7 fs_fsspec.exists(local_path)
      9 fs_fsspecstore = FsspecStore('file')
---> 10 fs_fsspecstore.exists(local_path)

File c:\Users\...\.venv\lib\site-packages\fsspec\asyn.py:118, in sync_wrapper.<locals>.wrapper(*args, **kwargs)
    115 @functools.wraps(func)
    116 def wrapper(*args, **kwargs):
    117     self = obj or args[0]
--> 118     return sync(self.loop, func, *args, **kwargs)

File c:\Users\...\.venv\lib\site-packages\fsspec\asyn.py:103, in sync(loop, func, timeout, *args, **kwargs)
    101     raise FSTimeoutError from return_result
    102 elif isinstance(return_result, BaseException):
--> 103     raise return_result
    104 else:
    105     return return_result

File c:\Users\...\.venv\lib\site-packages\fsspec\asyn.py:56, in _runner(event, coro, result, timeout)
     54     coro = asyncio.wait_for(coro, timeout=timeout)
     55 try:
---> 56     result[0] = await coro
     57 except Exception as ex:
     58     result[0] = ex

File c:\Users\...\.venv\lib\site-packages\fsspec\asyn.py:695, in AsyncFileSystem._exists(self, path, **kwargs)
    693 async def _exists(self, path, **kwargs):
    694     try:
--> 695         await self._info(path, **kwargs)
    696         return True
    697     except FileNotFoundError:

File c:\Users\...\.venv\lib\site-packages\obstore\fsspec.py:520, in FsspecStore._info(self, path, **_kwargs)
    515     pass
    517 # Ref: https://github.com/fsspec/s3fs/blob/01b9c4b838b81375093ae1d78562edf6bdc616ea/s3fs/core.py#L1471-L1492
    518 # We check to see if the path is a directory by attempting to list its
    519 # contexts. If anything is found, it is indeed a directory
--> 520 out = await self._ls(path, detail=True)
    521 if len(out) > 0:
    522     return {
    523         "name": f"{bucket}/{path_no_bucket}",
    524         "type": "directory",
    525         "size": 0,
    526     }

File c:\Users\...\.venv\lib\site-packages\obstore\fsspec.py:555, in FsspecStore._ls(self, path, detail, **_kwargs)
    552 bucket, path = self._split_path(path)
    553 store = self._construct_store(bucket)
--> 555 result = await obs.list_with_delimiter_async(store, path)
    556 objects = result["objects"]
    557 prefs = result["common_prefixes"]

GenericError: Generic LocalFileSystem error: Unable to walk dir: IO error for operation on C:\C:\dev\tmp: Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch. (os error 123)

Debug source:
Generic {
    store: "LocalFileSystem",
    source: UnableToWalkDir {
        source: Error {
            depth: 0,
            inner: Io {
                path: Some(
                    "C:\\C:\\dev\\tmp",
                ),
                err: Os {
                    code: 123,
                    kind: InvalidFilename,
                    message: "Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch.",
                },
            },
        },
    },
}

✅works fine

fs_fsspecstore.exists("C:/dev/tmp")

Metadata

Metadata

Assignees

No one assigned

    Labels

    fsspecPertaining to fsspec integration

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions