ASDF block API

[braingram]

Currently asdf doesn't provide any public API for interacting with ASDF blocks read from an ASDF file. There are some situations where this might be useful. One example is loading a block into a portion of a large array (without first loading the block into intermediate memory). Something like:

my_big_array = np.zeros((100, 2000, 2000))
for i, asdf_fn in enumerate(asdf_fns):  # assume len(asdf_fns) == 100
    with asdf.open(asdf_fn, lazy_load=True) as af:
        # let's assume af contains a single 2k x 2k block
        af.blocks[0].load(out=my_big_array[i])

I propose that we restrict this to "read-only" access at the moment (so AsdfFile.blocks will only be defined after a call to asdf.open(fn) and the block order and contents won't be modifiable). Minimally I propose that the API allow:

• finding the number of blocks in an file: len(af.blocks)
• access by index: blk = af.blocks[0]
• an ASDFBlock class (name tbd) that allows:
  • getting the block header (perhaps as a frozen dataclass): blk.header
  • getting the block header offset: blk.header_offset
  • getting the block data offset: blk.data_offset
  • getting the block data: blk.load()

Some items to discuss are:

• What read-only block information might be useful?
• What are some use cases for this new API?
• What are the pros and cons of having blk.load seed cached_data for the non-public block API?

[braingram]

Tagging @lgarrison @perrygreenfield @nden and @zacharyburnett for thoughts.

[zacharyburnett]

AsdfFile.blocks will only be defined after a call to asdf.open(fn)

how do we handle .blocks after the AsdfFile is .closed?

[braingram]

how do we handle .blocks after the AsdfFile is .closed?

Good question. Currently the private block manager is "closed" when the AsdfFile is closed:

asdf/asdf/_asdf.py

Line 553 in 8d9b721

self._blocks.close()

A decision would need to be made about what we support on a closed AsdfFile. We clear the tree on close. I think it may make sense to clear the blocks as well (so even if say the block header was loaded it would no longer be readable after the AsdfFile was closed).

[lgarrison]

Thanks for moving this forward! I think this kind of API would work fine with my use-case, which is very close to the example you gave. The only difference is that I usually access data by key in the tree, what's the best way to do that under this API? Would it be to get the block index from the tree/NDArrayType, then read the block?

[perrygreenfield]

Presumably a tool could be written to construct a dictionary of tree paths that map to each block?

[braingram]

Currently there's no public API for that (in part because there's no public block API). However, that's also not addressed by the above API proposal. It might be possible but there are some tricky corner cases (lazy tree where internal block-to-tree assignments won't occur until the tree content is accessed). Is there a particular use-case you had in mind?

[perrygreenfield]

I raised it since @lgarrison expressed the desire to index blocks by a key in the tree, and this could help do that. Wouldn't be possible to do the analysis on the raw yaml and bypass lazy tree machinery issues?

[braingram]

For NDArrayType some assumptions could be made about the source keyword in the yaml corresponding to the block. This doesn't hold for all block using objects (like zarr arrays).

[perrygreenfield]

I figured zarr cases wouldn't be covered (it's a little harder to think of good reasons to be able to do that anyway). It would rely on a convention or requirement to use the source keyword for most or all monolithic binary cases (e.g., object is represented by only one binary block)