-
Notifications
You must be signed in to change notification settings - Fork 33
ZEP9 (phase 1): add clarifications for extension naming #330
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 66 commits
Commits
Show all changes
67 commits
Select commit
Hold shift + click to select a range
65bc69f
Merge Davis proposal with ZEP0009
joshmoore b109fb7
Start changelog
joshmoore 4c0e494
Add definitions
joshmoore 05b4fa4
Fix definitions
joshmoore f9508d4
slightly longer change log
joshmoore 34ac282
New extensions section
joshmoore 16e34ca
Update array metadata section
joshmoore d2f6f9d
Update group metadata section
joshmoore 1d85e70
Clean the extension listing pages
joshmoore 43e3862
Also list no datatypes as defined
joshmoore c1accfe
Link more terms to extensions
joshmoore 454faaf
More crosslinks and identifier clarifications
joshmoore ef69ff1
add zarr-extensions repo
normanrz db7db15
Merge remote-tracking branch 'origin/main' into zep9-ext-naming
normanrz 3d448c1
Remove TODOs with PR and repo link
joshmoore e6200c8
Move 'core data types' to a subpage
joshmoore 0e0a03b
Clarify concept of 'core'
joshmoore 1600ee9
Unify listing of all extensions on subpages
joshmoore 429988a
Rename core/v3.0 to core/index
joshmoore c6fb150
Correct extensions table links
joshmoore 46630f7
Add 'core' to each of the subpages
joshmoore 20d6457
Simplify subpage headers
joshmoore a1d52b1
simplify reference to extensions in field definition
joshmoore 2b04661
simplify reference to extensions in field definition
joshmoore 1fa95e8
Implement suggestions from d-v-b and s/URI/URL/
joshmoore 95eef76
Move all v3 subdocs to index.rst
joshmoore fa0bdf6
Minor correction to a ref
joshmoore 3d86775
Add chunk key and grid subdocuments
joshmoore 7cfa69b
Clarify ext pts vs exts
joshmoore 9fdbd81
Clarify version policy applies independently to each page
joshmoore 7eef2b3
Make terms in ext list nicer
joshmoore 9e0f9d3
Drop versions from spec URIs
joshmoore 4e1bec8
Fix plurality of chunk grids page
joshmoore f2c977d
Unify all index pages into subdirectories
joshmoore d8c88ec
Catch a few last references of URIs rather than URLs
joshmoore 3844fc9
Improve changelog
joshmoore 447ad8c
Add Norman
joshmoore 6de00f1
Apply suggestions from code review
joshmoore a791ae5
fill_value
normanrz 4ba9e3e
versioning
normanrz 6e7da25
Make must_understand a section
joshmoore 8fd39d2
start with a lower-case letter
joshmoore 65e46f8
unify plurality of ext lists
joshmoore 90243d5
Improve URL description
joshmoore 73cce52
discourage top-level must_understand=false
joshmoore a0c3170
Add author guidance
joshmoore dbeb796
change extension example
normanrz 2724648
use namespaced names
rabernat 5c03a24
add URI as names section
rabernat 1ba5e41
Update docs/v3/core/index.rst
joshmoore c4a7215
Merge pull request #1 from rabernat/tweak-zep9-ext-naming
joshmoore 9eea54e
Minor updates to front matter
joshmoore 40732c0
Move fill_value to data_type section
joshmoore 7bb1cb1
Rename sections
joshmoore 7b1baff
Merge multiple edits
joshmoore aa9d87e
cleanup
joshmoore fac16fb
Fix objection typo
joshmoore b24440e
Minor fix
joshmoore 2861c84
Merge remote-tracking branch 'origin/main' into zep9-ext-naming
joshmoore 2d684a4
Apply suggestions from code review
joshmoore bf12d64
Introduction of tag:
joshmoore 97c483b
Update docs/v3/core/index.rst
joshmoore 6c1a027
Reduce to minimal change
joshmoore 8dc7796
Add feedback from ZSC
joshmoore 4ac0126
Correct whitespace
joshmoore dd90a3f
Remove confusingly redundant must_understand block from groups
joshmoore 7ee6d31
Make chunk-key-encoding info a warning
joshmoore File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
.. _chunk-grid-list: | ||
|
||
=========== | ||
Chunk Grids | ||
=========== | ||
|
||
The following documents specify chunk grids which SHOULD | ||
be implemented by all implementations. | ||
|
||
.. toctree:: | ||
:glob: | ||
:maxdepth: 1 | ||
:titlesonly: | ||
:caption: Contents: | ||
|
||
*/* | ||
|
||
Extensions | ||
---------- | ||
|
||
Registered chunk grid extensions can be found under | ||
`zarr-extensions::chunk-grids <https://github.com/zarr-developers/zarr-extensions/tree/main/chunk-grids>`_. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,117 @@ | ||
|
||
.. _regular-chunkgrid: | ||
|
||
================== | ||
Regular chunk grid | ||
================== | ||
|
||
Version: | ||
1.0 | ||
Specification URI: | ||
https://zarr-specs.readthedocs.io/en/latest/v3/chunk-grids/regular-grid/ | ||
Corresponding ZEP: | ||
`ZEP0001 — Zarr specification version 3 <https://zarr.dev/zeps/draft/ZEP0001.html>`_ | ||
Issue tracking: | ||
`GitHub issues <https://github.com/zarr-developers/zarr-specs/labels/chunk-grid>`_ | ||
Suggest an edit for this spec: | ||
`GitHub editor <https://github.com/zarr-developers/zarr-specs/blob/main/docs/v3/chunk-grids/regular-grid/index.rst>`_ | ||
|
||
Copyright 2020-Present Zarr core development team. This work | ||
is licensed under a `Creative Commons Attribution 3.0 Unported License | ||
<https://creativecommons.org/licenses/by/3.0/>`_. | ||
|
||
---- | ||
|
||
Abstract | ||
======== | ||
|
||
A regular grid is a type of grid where an array is divided into chunks | ||
such that each chunk is a hyperrectangle of the same shape. The | ||
dimensionality of the grid is the same as the dimensionality of the | ||
array. Each chunk in the grid can be addressed by a tuple of positive | ||
integers (`k`, `j`, `i`, ...) corresponding to the indices of the | ||
chunk along each dimension. | ||
|
||
Description | ||
=========== | ||
|
||
The origin element of a chunk has coordinates in the array space (`k` * | ||
`dz`, `j` * `dy`, `i` * `dx`, ...) where (`dz`, `dy`, `dx`, ...) are | ||
the chunk sizes along each dimension. | ||
Thus the origin element of the chunk at grid index (0, 0, 0, | ||
...) is at coordinate (0, 0, 0, ...) in the array space, i.e., the | ||
grid is aligned with the origin of the array. If the length of any | ||
array dimension is not perfectly divisible by the chunk length along | ||
the same dimension, then the grid will overhang the edge of the array | ||
space. | ||
|
||
The shape of the chunk grid will be (ceil(`z` / `dz`), ceil(`y` / | ||
`dy`), ceil(`x` / `dx`), ...) where (`z`, `y`, `x`, ...) is the array | ||
shape, "/" is the division operator and "ceil" is the ceiling | ||
function. For example, if a 3 dimensional array has shape (10, 200, | ||
3000), and has chunk shape (5, 20, 400), then the shape of the chunk | ||
grid will be (2, 10, 8), meaning that there will be 2 chunks along the | ||
first dimension, 10 along the second dimension, and 8 along the third | ||
dimension. | ||
|
||
.. list-table:: Regular Grid Example | ||
:header-rows: 1 | ||
|
||
* - Array Shape | ||
- Chunk Shape | ||
- Chunk Grid Shape | ||
- Notes | ||
* - (10, 200, 3000) | ||
- (5, 20, 400) | ||
- (2, 10, 8) | ||
- The grid does overhang the edge of the array on the 3rd dimension. | ||
|
||
An element of an array with coordinates (`c`, `b`, `a`, ...) will | ||
occur within the chunk at grid index (`c` // `dz`, `b` // `dy`, `a` // | ||
`dx`, ...), where "//" is the floor division operator. The element | ||
will have coordinates (`c` % `dz`, `b` % `dy`, `a` % `dx`, ...) within | ||
that chunk, where "%" is the modulo operator. For example, if a | ||
3 dimensional array has shape (10, 200, 3000), and has chunk shape | ||
(5, 20, 400), then the element of the array with coordinates (7, 150, 900) | ||
is contained within the chunk at grid index (1, 7, 2) and has coordinates | ||
(2, 10, 100) within that chunk. | ||
|
||
The store key corresponding to a given grid cell is determined based on the | ||
:ref:`array-metadata-chunk-key-encoding` member of the :ref:`array-metadata`. | ||
|
||
Note that this specification does not consider the case where the | ||
chunk grid and the array space are not aligned at the origin vertices | ||
of the array and the chunk at grid index (0, 0, 0, ...). However, | ||
extensions may define variations on the regular grid type | ||
such that the grid indices may include negative integers, and the | ||
origin element of the array may occur at an arbitrary position within | ||
any chunk, which is required to allow arrays to be extended by an | ||
arbitrary length in a "negative" direction along any dimension. | ||
|
||
.. note:: Chunks at the border of an array always have the full chunk size, even when | ||
the array only covers parts of it. For example, having an array with ``"shape": [30, 30]`` and | ||
``"chunk_shape": [16, 16]``, the chunk ``0,1`` would also contain unused values for the indices | ||
``0-16, 30-31``. When writing such chunks it is recommended to use the current fill value | ||
for elements outside the bounds of the array. | ||
|
||
|
||
|
||
Status of this document | ||
======================= | ||
|
||
ZEP0001 was accepted on May 15th, 2023 via https://github.com/zarr-developers/zarr-specs/issues/227. | ||
|
||
|
||
Document conventions | ||
==================== | ||
|
||
Conformance requirements are expressed with a combination of | ||
descriptive assertions and [RFC2119]_ terminology. The key words | ||
"MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", | ||
"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in the normative | ||
parts of this document are to be interpreted as described in | ||
[RFC2119]_. However, for readability, these words do not appear in all | ||
uppercase letters in this specification. | ||
|
||
All of the text of this specification is normative except sections | ||
explicitly marked as non-normative, examples, and notes. Examples in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,70 @@ | ||
.. _default-chunkkeyencoding: | ||
|
||
========================== | ||
Default chunk key encoding | ||
========================== | ||
|
||
Version: | ||
1.0 | ||
Specification URI: | ||
https://zarr-specs.readthedocs.io/en/latest/v3/chunk-key-encodings/default/ | ||
Corresponding ZEP: | ||
`ZEP0001 — Zarr specification version 3 <https://zarr.dev/zeps/draft/ZEP0001.html>`_ | ||
Issue tracking: | ||
`GitHub issues <https://github.com/zarr-developers/zarr-specs/labels/chunk-grid>`_ | ||
Suggest an edit for this spec: | ||
`GitHub editor <https://github.com/zarr-developers/zarr-specs/blob/main/docs/v3/chunk-key-encodings/default/index.rst>`_ | ||
|
||
Copyright 2020-Present Zarr core development team. This work | ||
is licensed under a `Creative Commons Attribution 3.0 Unported License | ||
<https://creativecommons.org/licenses/by/3.0/>`_. | ||
|
||
---- | ||
|
||
Description | ||
=========== | ||
|
||
The ``configuration`` object may contain one optional member, | ||
``separator``, which must be either ``"/"`` or ``"."``. If not specified, | ||
``separator`` defaults to ``"/"``. | ||
|
||
The key for a chunk with grid index (``k``, ``j``, ``i``, ...) is | ||
formed by taking the initial prefix ``c``, and appending for each dimension: | ||
|
||
- the ``separator`` character, followed by, | ||
|
||
- the ASCII decimal string representation of the chunk index within that dimension. | ||
|
||
For example, in a 3 dimensional array, with a separator of ``/`` the identifier | ||
for the chunk at grid index (1, 23, 45) is the string ``"c/1/23/45"``. With a | ||
separator of ``.``, the identifier is the string ``"c.1.23.45"``. The initial prefix | ||
``c`` ensures that metadata documents and chunks have separate prefixes. | ||
|
||
.. note:: A main difference with spec v2 is that the default chunk separator | ||
changed from ``.`` to ``/``, as in N5. This decreases the maximum number of | ||
items in hierarchical stores like directory stores. | ||
|
||
.. note:: Arrays may have 0 dimensions (when for example representing scalars), | ||
in which case the coordinate of a chunk is the empty tuple, and the chunk key | ||
will consist of the string ``c``. | ||
|
||
|
||
Status of this document | ||
======================= | ||
|
||
ZEP0001 was accepted on May 15th, 2023 via https://github.com/zarr-developers/zarr-specs/issues/227. | ||
|
||
|
||
Document conventions | ||
==================== | ||
|
||
Conformance requirements are expressed with a combination of | ||
descriptive assertions and [RFC2119]_ terminology. The key words | ||
"MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", | ||
"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in the normative | ||
parts of this document are to be interpreted as described in | ||
[RFC2119]_. However, for readability, these words do not appear in all | ||
uppercase letters in this specification. | ||
|
||
All of the text of this specification is normative except sections | ||
explicitly marked as non-normative, examples, and notes. Examples in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
.. _chunk-key-encoding-list: | ||
|
||
=================== | ||
Chunk Key Encodings | ||
=================== | ||
|
||
The following documents specify chunk key encodings which SHOULD | ||
be implemented by all implementations. | ||
|
||
.. toctree:: | ||
:glob: | ||
:maxdepth: 1 | ||
:titlesonly: | ||
:caption: Contents: | ||
|
||
*/* | ||
|
||
Extensions | ||
---------- | ||
|
||
Registered chunk grid extensions can be found under | ||
`zarr-extensions::chunk-key-encodings <https://github.com/zarr-developers/zarr-extensions/tree/main/chunk-key-encodings>`_. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,71 @@ | ||
.. _v2-chunkkeyencoding: | ||
|
||
===================== | ||
v2 chunk key encoding | ||
===================== | ||
|
||
Version: | ||
1.0 | ||
Specification URI: | ||
https://zarr-specs.readthedocs.io/en/latest/v3/chunk-key-encodings/v2/ | ||
Corresponding ZEP: | ||
`ZEP0001 — Zarr specification version 3 <https://zarr.dev/zeps/draft/ZEP0001.html>`_ | ||
Issue tracking: | ||
`GitHub issues <https://github.com/zarr-developers/zarr-specs/labels/chunk-grid>`_ | ||
Suggest an edit for this spec: | ||
`GitHub editor <https://github.com/zarr-developers/zarr-specs/blob/main/docs/v3/chunk-key-encodings/v2/index.rst>`_ | ||
|
||
Copyright 2020-Present Zarr core development team. This work | ||
is licensed under a `Creative Commons Attribution 3.0 Unported License | ||
<https://creativecommons.org/licenses/by/3.0/>`_. | ||
|
||
---- | ||
|
||
Description | ||
=========== | ||
|
||
The ``configuration`` object may contain one optional member, | ||
``separator``, which must be either ``"/"`` or ``"."``. If not specified, | ||
``separator`` defaults to ``"."``. | ||
|
||
The identifier for chunk with at least one dimension is formed by | ||
concatenating for each dimension: | ||
|
||
- the ASCII decimal string representation of the chunk index within that | ||
dimension, followed by | ||
|
||
- the ``separator`` character, except that it is omitted for the last | ||
dimension. | ||
|
||
For example, in a 3 dimensional array, with a separator of ``.`` the identifier | ||
for the chunk at grid index (1, 23, 45) is the string ``"1.23.45"``. With a | ||
separator of ``/``, the identifier is the string ``"1/23/45"``. | ||
|
||
For chunk grids with 0 dimensions, the single chunk has the key ``"0"``. | ||
|
||
.. note:: | ||
|
||
This encoding is intended only to allow existing v2 arrays to be | ||
converted to v3 without having to rename chunks. It is not recommended | ||
to be used when writing new arrays. | ||
joshmoore marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
|
||
Status of this document | ||
======================= | ||
|
||
ZEP0001 was accepted on May 15th, 2023 via https://github.com/zarr-developers/zarr-specs/issues/227. | ||
|
||
|
||
Document conventions | ||
==================== | ||
|
||
Conformance requirements are expressed with a combination of | ||
descriptive assertions and [RFC2119]_ terminology. The key words | ||
"MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", | ||
"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in the normative | ||
parts of this document are to be interpreted as described in | ||
[RFC2119]_. However, for readability, these words do not appear in all | ||
uppercase letters in this specification. | ||
|
||
All of the text of this specification is normative except sections | ||
explicitly marked as non-normative, examples, and notes. Examples in |
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.