NetCDF: One or more variable sizes violate format constraints

[3zero2]

When running this command:
python s5p-request.py L2__AER_AI --num-workers 1

I am getting the following error during the conversion to L3.

`Converting into L3 products

Launched 1 processes
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/wrfchem/anaconda3/envs/Sentinel5P/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/media/WRF_OUT_DATA/s5p-tools-2/s5p_tools/utils.py", line 68, in process_file
file_format="netcdf"
File "/home/wrfchem/anaconda3/envs/Sentinel5P/lib/python3.7/site-packages/harp/_harppy.py", line 1251, in export_product
raise CLibraryError()
harp._harppy.CLibraryError: NetCDF: One or more variable sizes violate format constraints (L3_data/S5P_NRTI_L3__AER_AI_20210712T074237_20210712T074737_19407_02_020200_20210712T080918.nc)
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "s5p-request.py", line 294, in
num_workers=args.num_workers,
File "s5p-request.py", line 142, in main
total=len(filenames),
File "/home/wrfchem/anaconda3/envs/Sentinel5P/lib/python3.7/site-packages/tqdm/std.py", line 1178, in iter
for obj in iterable:
File "/home/wrfchem/anaconda3/envs/Sentinel5P/lib/python3.7/multiprocessing/pool.py", line 748, in next
raise value
harp._harppy.CLibraryError: NetCDF: One or more variable sizes violate format constraints (L3_data/S5P_NRTI_L3__AER_AI_20210712T074237_20210712T074737_19407_02_020200_20210712T080918.nc)
`

Any idea what the cause might be please @bilelomrani1 ? Thank you.

[bilelomrani1]

I believe that this is due to a limitation of the NetCDF file format: each band cannot be larger than 2GB. When I developed these scripts, it was not an issue for my use case, but I reckon that this is a bit annoying.
One possible solution could be to use HDFS file format, I will leave this issue open and try to find some time to test this.