Decompression error #175
Description
Hi guys,
I have been trying to run the tutorial at https://abacusutils.readthedocs.io/en/latest/tutorials/analysis/zcv.html
Everything is smooth, until the last third step
python -m abacusnbody.hod.zcv.advect_fields --path2config $1
where I get
blosc_extension.error: Error -1 while decompressing data (Full message below.)
After writing a simple asdf toy script , I discovered that it might be due to memory issues (but not sure).
I was running with a configuration with nmesh=1152. Switching to half of it, nmesh=576, poses no problems. This was on Perlmutter, on a single Jupyter node, and Daint-Alps.
It should be connected to https://github.com/abacusorg/abacusutils/blob/main/abacusnbody/hod/zcv/ic_fields.py#L51
Playing with the setting might help (perhaps dealing with smaller chunks?). I will stick to nmesh=576 for now, but wanted to raise this (if not already known).
(lenscarf) odarwish@nid006155:~/colabdata> python -m abacusnbody.hod.zcv.ic_fields --path2config config.yaml
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/users/odarwish/abacusutils/abacusnbody/hod/zcv/ic_fields.py", line 512, in <module>
main(**args)
File "/users/odarwish/abacusutils/abacusnbody/hod/zcv/ic_fields.py", line 462, in main
compress_asdf(str(ic_fn), table, header)
File "/users/odarwish/abacusutils/abacusnbody/hod/zcv/ic_fields.py", line 57, in compress_asdf
print(f["data"]["dens"][...])
~~~~~~~~~~~~~~~~~^^^^^
File "/users/odarwish/lenscarf/lib/python3.12/site-packages/asdf/tags/core/ndarray.py", line 408, in operation
return getattr(self._make_array(), name)(*args)
^^^^^^^^^^^^^^^^^^
File "/users/odarwish/lenscarf/lib/python3.12/site-packages/asdf/tags/core/ndarray.py", line 282, in _make_array
data = self._data_callback(_attr="cached_data")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/users/odarwish/lenscarf/lib/python3.12/site-packages/asdf/_block/callback.py", line 40, in __call__
return getattr(read_blocks[self._index], _attr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/users/odarwish/lenscarf/lib/python3.12/site-packages/asdf/_block/reader.py", line 98, in cached_data
self._cached_data = self.data
^^^^^^^^^
File "/users/odarwish/lenscarf/lib/python3.12/site-packages/asdf/_block/reader.py", line 76, in data
data = self._data()
^^^^^^^^^^^^
File "/users/odarwish/lenscarf/lib/python3.12/site-packages/asdf/_block/io.py", line 225, in callback
data = read_block_data(fd, header, offset=data_offset, memmap=memmap)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/users/odarwish/lenscarf/lib/python3.12/site-packages/asdf/_block/io.py", line 151, in read_block_data
data = mcompression.decompress(fd, used_size, header["data_size"], compression)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/users/odarwish/lenscarf/lib/python3.12/site-packages/asdf/_compression.py", line 273, in decompress
len_decoded = decoder.decompress(blocks, out=buffer.data, **config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/users/odarwish/abacusutils/abacusnbody/data/asdf.py", line 165, in decompress
n_thisout = blosc.decompress_ptr(
^^^^^^^^^^^^^^^^^^^^^
File "/users/odarwish/lenscarf/lib/python3.12/site-packages/blosc/toplevel.py", line 669, in decompress_ptr
return _ext.decompress_ptr(bytes_like, address)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
blosc_extension.error: Error -1 while decompressing data