Dealing with inconsistent dimensions

[karlwx]

Hi everyone, I'm dealing with a rather frustrating issue with an operational script I'm running. The goal is to collect an entire model run worth of ensemble member data from the EPS for several variables, and open it in a single dataset. Typically, the script runs without a problem, but occasionally, what I suspect as an upstream data problem cause it to exit with an error.

Traceback (most recent call last):
  File "/home/meteo/kps5442/.conda/envs/wxworld1/lib/python3.11/site-packages/cfgrib/dataset.py", line 680, in build_dataset_components
    dict_merge(variables, coord_vars)
  File "/home/meteo/kps5442/.conda/envs/wxworld1/lib/python3.11/site-packages/cfgrib/dataset.py", line 611, in dict_merge
    raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='number' value=Variable(dimensions=('number',), data=array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
       35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50])) new_value=Variable(dimensions=('number',), data=array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
       36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50]))
skipping variable: paramId==228 shortName='tp'
Traceback (most recent call last):
  File "/home/meteo/kps5442/.conda/envs/wxworld1/lib/python3.11/site-packages/cfgrib/dataset.py", line 680, in build_dataset_components
    dict_merge(variables, coord_vars)
  File "/home/meteo/kps5442/.conda/envs/wxworld1/lib/python3.11/site-packages/cfgrib/dataset.py", line 611, in dict_merge
    raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='number' value=Variable(dimensions=('number',), data=array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
       35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50])) new_value=Variable(dimensions=('number',), data=array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 19,
       21, 22, 23, 24, 25, 26, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38,
       39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50]))
Traceback (most recent call last):
  File "/apple/s0/kps5442/operational/wxworld/eps.py", line 40, in <module>
    ds = xr.open_mfdataset(path, combine='nested', concat_dim='valid_time')
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/meteo/kps5442/.conda/envs/wxworld1/lib/python3.11/site-packages/xarray/backends/api.py", line 1053, in open_mfdataset
    combined = _nested_combine(
               ^^^^^^^^^^^^^^^^
  File "/home/meteo/kps5442/.conda/envs/wxworld1/lib/python3.11/site-packages/xarray/core/combine.py", line 359, in _nested_combine
    combined = _combine_nd(
               ^^^^^^^^^^^^
  File "/home/meteo/kps5442/.conda/envs/wxworld1/lib/python3.11/site-packages/xarray/core/combine.py", line 235, in _combine_nd
    combined_ids = _combine_all_along_first_dim(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/meteo/kps5442/.conda/envs/wxworld1/lib/python3.11/site-packages/xarray/core/combine.py", line 270, in _combine_all_along_first_dim
    new_combined_ids[new_id] = _combine_1d(
                               ^^^^^^^^^^^^
  File "/home/meteo/kps5442/.conda/envs/wxworld1/lib/python3.11/site-packages/xarray/core/combine.py", line 293, in _combine_1d
    combined = concat(
               ^^^^^^^
  File "/home/meteo/kps5442/.conda/envs/wxworld1/lib/python3.11/site-packages/xarray/core/concat.py", line 251, in concat
    return _dataset_concat(
           ^^^^^^^^^^^^^^^^
  File "/home/meteo/kps5442/.conda/envs/wxworld1/lib/python3.11/site-packages/xarray/core/concat.py", line 507, in _dataset_concat
    concat_over, equals, concat_dim_lengths = _calc_concat_over(
                                              ^^^^^^^^^^^^^^^^^^
  File "/home/meteo/kps5442/.conda/envs/wxworld1/lib/python3.11/site-packages/xarray/core/concat.py", line 408, in _calc_concat_over
    process_subset_opt(coords, "coords")
  File "/home/meteo/kps5442/.conda/envs/wxworld1/lib/python3.11/site-packages/xarray/core/concat.py", line 340, in process_subset_opt
    raise ValueError(
ValueError: 'surface' not present in all datasets and coords='different'. Either add 'surface' to datasets where it is missing or specify coords='minimal'.

It seems the issue is being caused by the fact that the variable 'number' (ensemble member number) is different between some of the files that are to be opened. There is one file per forecast hour, containing multiple members and variables. Ideally, this shouldn't happen, but occasionally data for one or two members is missing for a particular forecast hour, causing this problem.

We know that each file should have 50 members in the number dimension, numbered 1 to 50. Ideally, the behavior I want here is to still combine the datasets for all 50 members, and any missing members for a particular hour will have data that are arrays of NaNs with the same dimensions. Maybe this could be accomplished with a preprocessing step on each file, but I'm not sure how to go about that.

[TomNicholas]

There is a preprocess argument to open_mfdataset. Perhaps you could pass a function in there which checks the length along the number dimension and pads it if it's not the length you expect it to be?

[karlwx]

Yes, that's kind of what I was thinking. My question is more of how exactly to do this -- it's not clear to me how I can actually modify the dimension length and values.