xcube gen2 CLI doesn't append to the datacube when using multiple nc files as a source

[pierostz]

Hello,

I am trying to create datacubes from multiple netCDF4 files, based on this config and xcube gen2 CLI command. The dataset I am using is netCDF4 files downloaded from CAMS (Copernicus).
xcube gen2 config/no2/no2_l50.yml

The config is like this:

input_config:
  store_id: file
  data_id: l50.nc
  store_params:
    root: /app/data/no2

cube_config:
  variable_names: ["no2"]
  crs: "WGS84"
  bbox: [-24.95, 30.05, 44.95, 71.95]
  spatial_res: 0.1

output_config:
  store_id: file
  data_id: no2_l50.zarr
  store_params:
    root: /app/data/cubes
  replace: false

With the replace: true this command works fine but is not what I try to achieve. When I use replace: false I get this error: ERROR: Failed to write dataset '/app/data/cubes/no2_l50.zarr': path '' contains a group

The endgoal is:

1. Download netCDF4 datasets from CAMS.
2. Append the new datatsets to the datacube

I have also tried the xcube gen command, which I can see from the docs is deprecated, therefore the focus on the gen2 command. On that case though, what I experienced is that the netCDF files should be slices per time dimension and then use the -a parameter of the command to append. As I can see the time slicing needs to be done with the python API also.

Thanks

[forman]

Hi @pierostz, thanks for reporting.

Unfortunately, the xcube gen2 tool is in xcube still for backward compatibility with some of our project setups. We no longer use and maintain it. For new applications and projects we use and recommend the zappend tool and library. It has less features out-of-the box when compared to xcube gen2 but it is highly flexible and configurable. It has been made to robustly append dataset slices to Zarr datacubes with constant memory consumption including transaction management (which can be turned off).

I'm sorry for that confusion. Building really large (Zarr) data-cubes in a robust and reliable way remains a challenge and tools to support that will continue evolving.

[pierostz]

Hello @forman. Thanks for the reply.

So, to clarify. xcube gen and xcube gen2 are deprecated and I should use zappend to create and update the cubes?

Thanks

[forman]

@pierostz, yes, we invite you to give it a try :)

[pierostz]

@forman gave it a shot. Needed some preprocessing on the datasets I wanted from CAMS. Worked like a charm. I ditched gen and gen2 and replaced the functionality with zappend.

Thanks