Create read only copy if needed when opening a store path #3156

maxrjones · 2025-06-19T16:21:19Z

This PR makes a read-only copy of a store when using StorePath.open(mode="r").

See discussion in #3068. This augments #3068 (it still errors when attempting to open a writeable store path using a read-only store). This is an alternative to #3155.

Closes #3147

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/user-guide/*.rst
Changes documented as a new file in changes/
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

src/zarr/storage/_common.py

Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>

dstansby

Looks good to me. Requesting changes because:

This should get a release note
My old release note from when I added the error should be deleted

src/zarr/core/common.py

src/zarr/storage/_common.py

Co-authored-by: David Stansby <dstansby@gmail.com>

maxrjones · 2025-06-20T12:26:07Z

This should get a release note
My old release note from when I added the error should be deleted

Thanks, done in 10e50e4 (#3156)

This reverts commit 7ad760f.

src/zarr/storage/_common.py

TomNicholas · 2025-06-24T19:39:54Z

FYI I tried running xarray's zarr tests with this PR and while it fixes the basic read_only failure that #3068 caused (see for example this test, which passes with this PR), I still see a lot of other errors. I'm just trying to work out if those errors are caused by this PR or by zarr's dtypes refactor.

I also see a lot of things like this

FAILED xarray/tests/test_backends.py::TestZarrDictStore::test_warning_on_bad_chunks[2] - AssertionError: Got 1 unexpected warning(s): ["UserWarning('Store is not read-only but mode is 'r'. Creating a read-only copy. This behavior may change in the future with a more granular permissions model.'))"]

suggesting that xarray might need to swallow the warning being raised in this PR.

d-v-b · 2025-06-24T19:43:55Z

Why do we need to raise a warning here? Doesn't the new with_read_only method fully legitimize the creation of a new store in read-only mode?

TomNicholas · 2025-06-24T19:46:00Z

I agree, this isn't something to warn for, it's just a change of behaviour to be documented.

rabernat · 2025-06-24T20:00:33Z

I agree, this isn't something to warn for, it's just a change of behaviour to be documented.

👍 to this. FWIW I think we have been way to eager to issue warnings in Zarr. No warning is needed here.

TomNicholas · 2025-06-24T20:01:11Z

I'm just trying to work out if those errors are caused by this PR or by zarr's dtypes refactor.

After talking to @ianhi I think those other errors are related to the dtypes changes rather than to the changes in this PR.

maxrjones · 2025-06-24T20:13:14Z

I'm just trying to work out if those errors are caused by this PR or by zarr's dtypes refactor.

After talking to @ianhi I think those other errors are related to the dtypes changes rather than to the changes in this PR.

Yes they are definitely unrelated to this and instead are relevant to the dtypes changes.

Co-authored-by: Tom Nicholas <tom@earthmover.io>

maxrjones · 2025-06-24T20:41:46Z

Why do we need to raise a warning here? Doesn't the new with_read_only method fully legitimize the creation of a new store in read-only mode?

thanks for this suggestion, Davis, and the affirmations, Tom and Ryan. The PR no longer includes a warning. My decision to initially include one was influenced by the prevalence of warnings elsewhere so it's helpful to have that general correction.

rabernat

It would be good to get coverage back up before merging.

David's comments have clearly been addressed, he's on vacation and can't approve. GH won't let us merge with pending changes requested..

d-v-b · 2025-06-30T09:02:18Z

@meeseeksdev backport to 3.0.9

lumberbot-app · 2025-06-30T09:02:26Z

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

Checkout backport branch and update it.

git checkout 3.0.9
git pull

Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 5731c6c88ef0c5a3cb7f9701a7fbcee2814febf3

You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #3156: Create read only copy if needed when opening a store path'

Push to a named branch:

git push YOURFORK 3.0.9:auto-backport-of-pr-3156-on-3.0.9

Create a PR against branch 3.0.9, I would have named this PR:

"Backport PR #3156 on branch 3.0.9 (Create read only copy if needed when opening a store path)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

d-v-b · 2025-06-30T09:17:45Z

@meeseeksdev backport to 3.0.9

lumberbot-app · 2025-06-30T09:17:56Z

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

Checkout backport branch and update it.

git checkout 3.0.9
git pull

Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 5731c6c88ef0c5a3cb7f9701a7fbcee2814febf3

You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #3156: Create read only copy if needed when opening a store path'

Push to a named branch:

git push YOURFORK 3.0.9:auto-backport-of-pr-3156-on-3.0.9

Create a PR against branch 3.0.9, I would have named this PR:

"Backport PR #3156 on branch 3.0.9 (Create read only copy if needed when opening a store path)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

…opers#3156) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening Co-authored-by: Tom Nicholas <tom@earthmover.io> * Improve code coverage --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> (cherry picked from commit 5731c6c)

) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py * Update src/zarr/storage/_common.py * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening * Improve code coverage --------- (cherry picked from commit 5731c6c) Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io>

* Create read only copy if needed when opening a store path (#3156) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening Co-authored-by: Tom Nicholas <tom@earthmover.io> * Improve code coverage --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> (cherry picked from commit 5731c6c) * release notes --------- Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io>

* Enable tests on 3.0.x branch (#3135) * Changelog for 3.0.9 * Backport PR #3149: Add GroupNotFound error to API docs (#3179) Co-authored-by: David Stansby <dstansby@gmail.com> * Backport PR #3140: Suppress FileNotFoundError when deleting keys in the obstore adapter (#3180) Co-authored-by: Lukas Bindreiter <lukas.bindreiter@tilebox.com> * Backport PR #3138: Add with_read_only() convenience method to store (#3181) Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> * Create read only copy if needed when opening a store path (#3156) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening Co-authored-by: Tom Nicholas <tom@earthmover.io> * Improve code coverage --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> (cherry picked from commit 5731c6c) * Create read only copy if needed when opening a store path (#3156) (#3182) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py * Update src/zarr/storage/_common.py * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening * Improve code coverage --------- (cherry picked from commit 5731c6c) Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> * Remove breaking check about `auto_mkdir` for FSSpecStore (#3193) * Remove breaking check from _make_async * Update expected error * Change import structure to protect against AttributeError * changelog * add test to ensure that we can create a read-only copy of the store with auto_mkdir=False * only test if the async wrapper is available --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> (cherry picked from commit 5a24487) * Remove breaking check about `auto_mkdir` for FSSpecStore (#3193) (#3203) * Remove breaking check from _make_async * Update expected error * Change import structure to protect against AttributeError * changelog * add test to ensure that we can create a read-only copy of the store with auto_mkdir=False * only test if the async wrapper is available --------- (cherry picked from commit 5a24487) Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> * Add missing import for AsyncFileSystemWrapper for `_make_async` in `_fsspec.py` (#3195) * Add missing import for AsyncFileSystemWrapper in `_fsspec.py` * Add missing changelog entry for AsyncFileSystemWrapper import fix * Move AsyncFileSystemWrapper import past the version check in `_fsspec.py` * Add newline after AsyncFileSystemWrapper import in `_fsspec.py` * Simplify import statement for AsyncFileSystemWrapper in `_fsspec.py` --------- Co-authored-by: Altay Sansal <altay.sansal@tgs.com> (cherry picked from commit 97aa42f) * Auto backport of pr 3195 on 3.0.10 (#3204) * Remove breaking check about `auto_mkdir` for FSSpecStore (#3193) * Remove breaking check from _make_async * Update expected error * Change import structure to protect against AttributeError * changelog * add test to ensure that we can create a read-only copy of the store with auto_mkdir=False * only test if the async wrapper is available --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> (cherry picked from commit 5a24487) * Add missing import for AsyncFileSystemWrapper for `_make_async` in `_fsspec.py` (#3195) * Add missing import for AsyncFileSystemWrapper in `_fsspec.py` * Add missing changelog entry for AsyncFileSystemWrapper import fix * Move AsyncFileSystemWrapper import past the version check in `_fsspec.py` * Add newline after AsyncFileSystemWrapper import in `_fsspec.py` * Simplify import statement for AsyncFileSystemWrapper in `_fsspec.py` --------- Co-authored-by: Altay Sansal <altay.sansal@tgs.com> (cherry picked from commit 97aa42f) --------- Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: Altay Sansal <tasansal@users.noreply.github.com> Co-authored-by: Altay Sansal <altay.sansal@tgs.com> * 3.0.9 release notes (#3183) * Create read only copy if needed when opening a store path (#3156) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening Co-authored-by: Tom Nicholas <tom@earthmover.io> * Improve code coverage --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> (cherry picked from commit 5731c6c) * release notes --------- Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> * release notes --------- Co-authored-by: Lumberbot (aka Jack) <39504233+meeseeksmachine@users.noreply.github.com> Co-authored-by: Lukas Bindreiter <lukas.bindreiter@tilebox.com> Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> Co-authored-by: Altay Sansal <tasansal@users.noreply.github.com> Co-authored-by: Altay Sansal <altay.sansal@tgs.com>

Create read only copy if needed when opening a store path

6f52bfb

github-actions bot added the needs release notes Automatically applied to PRs which haven't added release notes label Jun 19, 2025

Add ValueError to Raises section

737ff9b

maxrjones commented Jun 19, 2025

View reviewed changes

src/zarr/storage/_common.py Show resolved Hide resolved

maxrjones commented Jun 19, 2025

View reviewed changes

src/zarr/storage/_common.py Show resolved Hide resolved

maxrjones commented Jun 19, 2025

View reviewed changes

src/zarr/storage/_common.py Show resolved Hide resolved

Update expected warning

957cd64

d-v-b reviewed Jun 19, 2025

View reviewed changes

src/zarr/storage/_common.py Outdated Show resolved Hide resolved

d-v-b reviewed Jun 19, 2025

View reviewed changes

src/zarr/storage/_common.py Outdated Show resolved Hide resolved

maxrjones and others added 2 commits June 19, 2025 13:19

Update src/zarr/storage/_common.py

23f4376

Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>

Use ANY_ACCESS_MODE

a42dc7c

maxrjones mentioned this pull request Jun 19, 2025

Only warn if opening writeable store with mode='r' #3155

Closed

dstansby previously requested changes Jun 20, 2025

View reviewed changes

src/zarr/core/common.py Show resolved Hide resolved

src/zarr/storage/_common.py Show resolved Hide resolved

src/zarr/storage/_common.py Outdated Show resolved Hide resolved

src/zarr/storage/_common.py Outdated Show resolved Hide resolved

maxrjones and others added 3 commits June 20, 2025 08:21

Update src/zarr/storage/_common.py

aba4919

Co-authored-by: David Stansby <dstansby@gmail.com>

Update src/zarr/storage/_common.py

0129458

Co-authored-by: David Stansby <dstansby@gmail.com>

Update changes

10e50e4

github-actions bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Jun 20, 2025

maxrjones added 3 commits June 20, 2025 08:26

Merge branch 'main' into maybe-copy-store

15becdf

Try using get_args on definition

7ad760f

Revert "Try using get_args on definition"

a1949f1

This reverts commit 7ad760f.

maxrjones requested a review from dstansby June 20, 2025 13:41

Add test

202d606

TomNicholas reviewed Jun 24, 2025

View reviewed changes

src/zarr/storage/_common.py Outdated Show resolved Hide resolved

src/zarr/storage/_common.py Outdated Show resolved Hide resolved

src/zarr/storage/_common.py Show resolved Hide resolved

Merge branch 'main' into maybe-copy-store

4d8e7ad

maxrjones and others added 3 commits June 24, 2025 16:35

Remove warning

4e099e4

Apply suggestion for try; except shortening

2616b2f

Co-authored-by: Tom Nicholas <tom@earthmover.io>

Merge branch 'main' into maybe-copy-store

0b1e31b

rabernat approved these changes Jun 24, 2025

View reviewed changes

Improve code coverage

07bb607

rabernat merged commit 5731c6c into zarr-developers:main Jun 24, 2025
30 checks passed

maxrjones deleted the maybe-copy-store branch June 25, 2025 00:09

d-v-b added this to the 3.0.9 milestone Jun 30, 2025

lumberbot-app bot added the Still Needs Manual Backport label Jun 30, 2025

d-v-b removed the Still Needs Manual Backport label Jun 30, 2025

d-v-b mentioned this pull request Jun 30, 2025

3.0.9 release notes #3183

Merged

This was referenced Jun 30, 2025

⚠️ Nightly upstream-dev CI failed ⚠️ pydata/xarray#10261

Closed

ODD PI 25.3 Objective 5: Foundational Zarr-Python and Xarray Contributions NASA-IMPACT/veda-odd#165

Closed

maxrjones mentioned this pull request Jul 6, 2025

Use a release checklist and narrative announcement for breaking changes #3209

Open

Uh oh!

Create read only copy if needed when opening a store path #3156

Create read only copy if needed when opening a store path #3156

Uh oh!

Conversation

maxrjones commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dstansby left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxrjones commented Jun 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TomNicholas commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

d-v-b commented Jun 24, 2025

Uh oh!

TomNicholas commented Jun 24, 2025

Uh oh!

rabernat commented Jun 24, 2025

Uh oh!

TomNicholas commented Jun 24, 2025

Uh oh!

maxrjones commented Jun 24, 2025

Uh oh!

maxrjones commented Jun 24, 2025

Uh oh!

rabernat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

d-v-b commented Jun 30, 2025

Uh oh!

lumberbot-app bot commented Jun 30, 2025

Uh oh!

d-v-b commented Jun 30, 2025

Uh oh!

lumberbot-app bot commented Jun 30, 2025

Uh oh!

Uh oh!

maxrjones commented Jun 19, 2025 •

edited

Loading

TomNicholas commented Jun 24, 2025 •

edited

Loading