Add missing import for AsyncFileSystemWrapper for `_make_async` in `_fsspec.py` #3195

tasansal · 2025-07-02T14:43:31Z

When using FSSpec's memory filesystem, the new _make_async function falls back to using below:

Line 71 in baabf08

    
           return fsspec.implementations.asyn_wrapper.AsyncFileSystemWrapper(fs, asynchronous=True)

However, this is not imported and results in an AttributeError. This is expected since the submodule is not imported and we are trying to call it from fsspec module itself.

Added explicit import of the submodule.

Reproducer:

import zarr
zarr.open_group("memory://group.zarr")

Python 3.13
zarr-python version 3.0.9
fsspec version 2025.5.1

I'm not sure if any unit tests are necessary here since its a simple fix of a minor oversight. May I suggest doing a 3.0.9 post-release and yanking the 3.0.9 ASAP.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/user-guide/*.rst
Changes documented as a new file in changes/
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

tasansal · 2025-07-02T15:10:07Z

Anything to worry about in the failing tests or are those the usual flaky tests?

joshmoore · 2025-07-02T15:12:57Z

FAILED tests/test_store/test_fsspec.py::test_wrap_sync_filesystem_raises - AssertionError: Regex pattern did not match.
 Regex: 'The filesystem .*'
 Input: "No module named 'fsspec.implementations.asyn_wrapper'"

tasansal · 2025-07-02T15:17:32Z

FAILED tests/test_store/test_fsspec.py::test_wrap_sync_filesystem_raises - AssertionError: Regex pattern did not match.
 Regex: 'The filesystem .*'
 Input: "No module named 'fsspec.implementations.asyn_wrapper'"

Ok I think needs to be imported past the fsspec version condition.

….py`

maxrjones

I suggested the slightly more targeted approach that I included in #3193.

src/zarr/storage/_fsspec.py

d-v-b · 2025-07-02T16:29:49Z

i feel like this should have been caught in our tests. why wasn't it?

maxrjones · 2025-07-02T17:24:27Z

i feel like this should have been caught in our tests. why wasn't it?

I think because the lines below in the tests mean that the module gets loaded, whereas in a user's workflow the module may not already be loaded since there are the modules are not imported in fsspec/__init__.py and fsspec/implementation/__init__.py.

zarr-python/tests/test_store/test_fsspec.py

Line 301 in baabf08

from fsspec.implementations.asyn_wrapper import AsyncFileSystemWrapper
zarr-python/tests/test_store/test_fsspec.py

Line 325 in baabf08

from fsspec.implementations.asyn_wrapper import AsyncFileSystemWrapper
zarr-python/tests/test_store/test_fsspec.py

Line 411 in baabf08

from fsspec.implementations.asyn_wrapper import AsyncFileSystemWrapper

tasansal · 2025-07-02T18:24:17Z

Yea having the lazy import inside function could cause some issues like this. One way to fix that is module level import conditional on fsspec version etc; but it also gets ugly :)

tasansal · 2025-07-02T19:32:33Z

@joshmoore seems like this is ready to merge. What are your thoughts on 3.0.9 post-release and yanking 3.0.9?

d-v-b · 2025-07-02T20:53:35Z

IMO we don't need to yank 3.0.9, but we should issue a 3.0.10 release very soon

d-v-b · 2025-07-03T17:04:52Z

@meeseeksdev backport to 3.0.10

lumberbot-app · 2025-07-03T17:04:59Z

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

Checkout backport branch and update it.

git checkout 3.0.10
git pull

Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 97aa42f551c2e970f66ee629cb31ce426e659b7b

You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #3195: Add missing import for AsyncFileSystemWrapper for `_make_async` in `_fsspec.py`'

Push to a named branch:

git push YOURFORK 3.0.10:auto-backport-of-pr-3195-on-3.0.10

Create a PR against branch 3.0.10, I would have named this PR:

"Backport PR #3195 on branch 3.0.10 (Add missing import for AsyncFileSystemWrapper for _make_async in _fsspec.py)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

…fsspec.py` (zarr-developers#3195) * Add missing import for AsyncFileSystemWrapper in `_fsspec.py` * Add missing changelog entry for AsyncFileSystemWrapper import fix * Move AsyncFileSystemWrapper import past the version check in `_fsspec.py` * Add newline after AsyncFileSystemWrapper import in `_fsspec.py` * Simplify import statement for AsyncFileSystemWrapper in `_fsspec.py` --------- Co-authored-by: Altay Sansal <altay.sansal@tgs.com> (cherry picked from commit 97aa42f)

* Remove breaking check about `auto_mkdir` for FSSpecStore (#3193) * Remove breaking check from _make_async * Update expected error * Change import structure to protect against AttributeError * changelog * add test to ensure that we can create a read-only copy of the store with auto_mkdir=False * only test if the async wrapper is available --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> (cherry picked from commit 5a24487) * Add missing import for AsyncFileSystemWrapper for `_make_async` in `_fsspec.py` (#3195) * Add missing import for AsyncFileSystemWrapper in `_fsspec.py` * Add missing changelog entry for AsyncFileSystemWrapper import fix * Move AsyncFileSystemWrapper import past the version check in `_fsspec.py` * Add newline after AsyncFileSystemWrapper import in `_fsspec.py` * Simplify import statement for AsyncFileSystemWrapper in `_fsspec.py` --------- Co-authored-by: Altay Sansal <altay.sansal@tgs.com> (cherry picked from commit 97aa42f) --------- Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: Altay Sansal <tasansal@users.noreply.github.com> Co-authored-by: Altay Sansal <altay.sansal@tgs.com>

* Enable tests on 3.0.x branch (#3135) * Changelog for 3.0.9 * Backport PR #3149: Add GroupNotFound error to API docs (#3179) Co-authored-by: David Stansby <dstansby@gmail.com> * Backport PR #3140: Suppress FileNotFoundError when deleting keys in the obstore adapter (#3180) Co-authored-by: Lukas Bindreiter <lukas.bindreiter@tilebox.com> * Backport PR #3138: Add with_read_only() convenience method to store (#3181) Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> * Create read only copy if needed when opening a store path (#3156) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening Co-authored-by: Tom Nicholas <tom@earthmover.io> * Improve code coverage --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> (cherry picked from commit 5731c6c) * Create read only copy if needed when opening a store path (#3156) (#3182) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py * Update src/zarr/storage/_common.py * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening * Improve code coverage --------- (cherry picked from commit 5731c6c) Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> * Remove breaking check about `auto_mkdir` for FSSpecStore (#3193) * Remove breaking check from _make_async * Update expected error * Change import structure to protect against AttributeError * changelog * add test to ensure that we can create a read-only copy of the store with auto_mkdir=False * only test if the async wrapper is available --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> (cherry picked from commit 5a24487) * Remove breaking check about `auto_mkdir` for FSSpecStore (#3193) (#3203) * Remove breaking check from _make_async * Update expected error * Change import structure to protect against AttributeError * changelog * add test to ensure that we can create a read-only copy of the store with auto_mkdir=False * only test if the async wrapper is available --------- (cherry picked from commit 5a24487) Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> * Add missing import for AsyncFileSystemWrapper for `_make_async` in `_fsspec.py` (#3195) * Add missing import for AsyncFileSystemWrapper in `_fsspec.py` * Add missing changelog entry for AsyncFileSystemWrapper import fix * Move AsyncFileSystemWrapper import past the version check in `_fsspec.py` * Add newline after AsyncFileSystemWrapper import in `_fsspec.py` * Simplify import statement for AsyncFileSystemWrapper in `_fsspec.py` --------- Co-authored-by: Altay Sansal <altay.sansal@tgs.com> (cherry picked from commit 97aa42f) * Auto backport of pr 3195 on 3.0.10 (#3204) * Remove breaking check about `auto_mkdir` for FSSpecStore (#3193) * Remove breaking check from _make_async * Update expected error * Change import structure to protect against AttributeError * changelog * add test to ensure that we can create a read-only copy of the store with auto_mkdir=False * only test if the async wrapper is available --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> (cherry picked from commit 5a24487) * Add missing import for AsyncFileSystemWrapper for `_make_async` in `_fsspec.py` (#3195) * Add missing import for AsyncFileSystemWrapper in `_fsspec.py` * Add missing changelog entry for AsyncFileSystemWrapper import fix * Move AsyncFileSystemWrapper import past the version check in `_fsspec.py` * Add newline after AsyncFileSystemWrapper import in `_fsspec.py` * Simplify import statement for AsyncFileSystemWrapper in `_fsspec.py` --------- Co-authored-by: Altay Sansal <altay.sansal@tgs.com> (cherry picked from commit 97aa42f) --------- Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: Altay Sansal <tasansal@users.noreply.github.com> Co-authored-by: Altay Sansal <altay.sansal@tgs.com> * 3.0.9 release notes (#3183) * Create read only copy if needed when opening a store path (#3156) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening Co-authored-by: Tom Nicholas <tom@earthmover.io> * Improve code coverage --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> (cherry picked from commit 5731c6c) * release notes --------- Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> * release notes --------- Co-authored-by: Lumberbot (aka Jack) <39504233+meeseeksmachine@users.noreply.github.com> Co-authored-by: Lukas Bindreiter <lukas.bindreiter@tilebox.com> Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> Co-authored-by: Altay Sansal <tasansal@users.noreply.github.com> Co-authored-by: Altay Sansal <altay.sansal@tgs.com>

Add missing import for AsyncFileSystemWrapper in _fsspec.py

9598797

github-actions bot added the needs release notes Automatically applied to PRs which haven't added release notes label Jul 2, 2025

tasansal changed the title ~~Add missing import for AsyncFileSystemWrapper in _fsspec.py~~ Add missing import for AsyncFileSystemWrapper for _make_async in _fsspec.py Jul 2, 2025

Add missing changelog entry for AsyncFileSystemWrapper import fix

d9f7aef

github-actions bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Jul 2, 2025

tasansal marked this pull request as ready for review July 2, 2025 14:49

tasansal mentioned this pull request Jul 2, 2025

[v1] Update dependencies to latest TGSAI/mdio-python#567

Merged

Altay Sansal added 2 commits July 2, 2025 10:18

Move AsyncFileSystemWrapper import past the version check in `_fsspec…

965f962

….py`

Add newline after AsyncFileSystemWrapper import in _fsspec.py

e0999b2

dgegen mentioned this pull request Jul 2, 2025

Issue Opening Files via SFTP with fsspec in Zarr v3 #3196

Open

maxrjones reviewed Jul 2, 2025

View reviewed changes

src/zarr/storage/_fsspec.py Outdated Show resolved Hide resolved

src/zarr/storage/_fsspec.py Outdated Show resolved Hide resolved

Simplify import statement for AsyncFileSystemWrapper in _fsspec.py

a3b97ac

tasansal requested a review from maxrjones July 2, 2025 19:33

maxrjones approved these changes Jul 3, 2025

View reviewed changes

d-v-b approved these changes Jul 3, 2025

View reviewed changes

d-v-b merged commit 97aa42f into zarr-developers:main Jul 3, 2025
30 checks passed

d-v-b added this to the 3.0.10 milestone Jul 3, 2025

lumberbot-app bot added the Still Needs Manual Backport label Jul 3, 2025

d-v-b removed the Still Needs Manual Backport label Jul 3, 2025

maxrjones mentioned this pull request Jul 6, 2025

Use a release checklist and narrative announcement for breaking changes #3209

Open

tasansal mentioned this pull request Jul 7, 2025

Remove Zarr version pin because bug is fixed TGSAI/mdio-python#569

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add missing import for AsyncFileSystemWrapper for `_make_async` in `_fsspec.py` #3195

Add missing import for AsyncFileSystemWrapper for `_make_async` in `_fsspec.py` #3195

Uh oh!

tasansal commented Jul 2, 2025 •

edited

Loading

Uh oh!

tasansal commented Jul 2, 2025

Uh oh!

joshmoore commented Jul 2, 2025

Uh oh!

tasansal commented Jul 2, 2025 •

edited

Loading

Uh oh!

maxrjones left a comment

Uh oh!

Uh oh!

Uh oh!

d-v-b commented Jul 2, 2025

Uh oh!

maxrjones commented Jul 2, 2025 •

edited

Loading

Uh oh!

tasansal commented Jul 2, 2025

Uh oh!

tasansal commented Jul 2, 2025

Uh oh!

d-v-b commented Jul 2, 2025

Uh oh!

Uh oh!

d-v-b commented Jul 3, 2025

Uh oh!

lumberbot-app bot commented Jul 3, 2025

Uh oh!

Uh oh!

Uh oh!

Add missing import for AsyncFileSystemWrapper for _make_async in _fsspec.py #3195

Add missing import for AsyncFileSystemWrapper for _make_async in _fsspec.py #3195

Uh oh!

Conversation

tasansal commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tasansal commented Jul 2, 2025

Uh oh!

joshmoore commented Jul 2, 2025

Uh oh!

tasansal commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maxrjones left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

d-v-b commented Jul 2, 2025

Uh oh!

maxrjones commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tasansal commented Jul 2, 2025

Uh oh!

tasansal commented Jul 2, 2025

Uh oh!

d-v-b commented Jul 2, 2025

Uh oh!

Uh oh!

d-v-b commented Jul 3, 2025

Uh oh!

lumberbot-app bot commented Jul 3, 2025

Uh oh!

Uh oh!

Add missing import for AsyncFileSystemWrapper for `_make_async` in `_fsspec.py` #3195

Add missing import for AsyncFileSystemWrapper for `_make_async` in `_fsspec.py` #3195

tasansal commented Jul 2, 2025 •

edited

Loading

tasansal commented Jul 2, 2025 •

edited

Loading

maxrjones commented Jul 2, 2025 •

edited

Loading