specify bool as 1 byte

[joshmoore]

Should we specify that this is 1 byte?

   * - ``bool``
     - Boolean (1 byte)

This would be consistent with NumPy and existing implementations

Also it would clarify that we don't mean packed bits

Originally posted by @jakirkham in #330 (comment)

[jbms]

No, because the in-memory representation is up to the implementation and the stored representation in chunks is up to the array to bytes codec. The bytes codec specifies 1 byte and the packbits codec that I proposed in zarr-extensions specifies 1 bit.

[jakirkham]

In-memory representations are always up to implementations

The point of the spec is to ensure that when we write out data others understand what we have written and can read it

The intended functional behavior of bool currently is 1 byte. That is how it is already used in the wild

The Bytes codec is merely capturing how things often are represented in memory already and so how they would look unchanged when going to disk

A packed bits codec is certainly a valid thing to discuss. We have PackBits in Numcodecs. However its output is not bool. Its output is raw bits, which is also defined later

zarr-specs/docs/v3/data-types/index.rst

Lines 46 to 47 in 1d0989e

	* - ``r*`` (Optional)
	- raw bits, variable size given by ``*``, limited to be a multiple of 8

[jbms]

In-memory representations are always up to implementations

The point of the spec is to ensure that when we write out data others understand what we have written and can read it

The intended functional behavior of bool currently is 1 byte. That is how it is already used in the wild

I'm not sure I understand what it means to say that it is "1 byte" if we aren't talking about either the in-memory representation or the on-disk representation.

The Bytes codec is merely capturing how things often are represented in memory already and so how they would look unchanged when going to disk

That is a possible interpretation and indeed true (module endianness) for the existing implementations I'm aware of. But it is also quite possible that an implementation would choose to always represent bool in packed representation in memory (and in fact this is likely what I'll do for Neuroglancer where memory efficiency, especially gpu memory efficiency, is particularly critical) in which case bytes would require an encoding/decoding step while packbits would not.

A packed bits codec is certainly a valid thing to discuss. We have PackBits in Numcodecs. However its output is not bool. Its output is raw bits, which is also defined later

That is an interesting way of looking at packbits that I hadn't considered --- I had considered it to be an array -> bytes codec that supports a variety of different data types, including bool. Alternatively you could say that packbits only works with rawbits data types (which have been relaxed to allow non-multiple-of-8 sizes) and you need to first use an array -> array codec (not sure what this would be called) to convert other data types to rawbits, and then use an array -> bytes codec that supports not-multiple-of-8-bits rawbits data types, perhaps called packbits . I'm not sure about the relative merits of these two approaches. Separating the conversion to bits from the encoding into a chunk in principle provides greater flexibility as far as supporting multiple encoding strategies (e.g. maybe a shuffle bits/transpose step) but may also pose greater implementation challenges.

Note: This is my packbits proposal for zarr v3 (https://github.com/zarr-developers/zarr-extensions/pull/3/files#diff-8c721f8bc1d30cef0875281a1a9f922d762a0998f1601cd05022e544ed8df271), which is designed to be compatible with the numcodecs packbits codec.

zarr-specs/docs/v3/data-types/index.rst

Lines 46 to 47 in 1d0989e

• • r* (Optional)
  • raw bits, variable size given by *, limited to be a multiple of 8