open_mfdataset not returning all data

[BrianBlanton]

Hi, I have a few grib2 files and am trying to use open_mfdataset to read into one variable. A preprocess function makes a new time coord, along which the concatting should happen.

But the resulting DataSet only contains the last file's data. The short code and the output of printing the datasets is below. Seems like this should work.

Has anyone seen similar behavior, or otherwise see something wrong? I've posted the 2 grib files here, if anyone would like to try them. multi_reanal.ecg_4m.hs.197901.grb2, multi_reanal.ecg_4m.hs.197902.grb

import xarray as xr

def preprocess(ds):
    dsnew = ds.copy()
    dsnew['time']=(ds.step+ds.time)
    print("\n")
    print(dsnew)
    print("\n")
    return dsnew

files=['multi_reanal.ecg_4m.hs.197901.grb2', 'multi_reanal.ecg_4m.hs.197902.grb2']

ds=xr.open_mfdataset(files, engine='cfgrib', preprocess=preprocess,  concat_dim='step', parallel=False)
print("\n\nReturned ds from open_mfdataset:\n")
print(ds)

<xarray.Dataset>
Dimensions:     (latitude: 481, longitude: 586, step: 249)
Coordinates:
    time        (step) datetime64[ns] 1979-01-01 ... 1979-02-01
  * step        (step) timedelta64[ns] 0 days 00:00:00 ... 31 days 00:00:00
  * latitude    (latitude) float64 47.0 46.93 46.87 46.8 ... 15.13 15.07 15.0
  * longitude   (longitude) float64 261.0 261.1 261.1 ... 299.9 299.9 300.0
    valid_time  (step) datetime64[ns] dask.array<chunksize=(249,), meta=np.ndarray>
Data variables:
    swh         (step, latitude, longitude) float32 dask.array<chunksize=(249, 481, 586), meta=np.ndarray>
Attributes:
    GRIB_edition:            2
...


<xarray.Dataset>
Dimensions:     (latitude: 481, longitude: 586, step: 225)
Coordinates:
    time        (step) datetime64[ns] 1979-02-01 ... 1979-03-01
  * step        (step) timedelta64[ns] 0 days 00:00:00 ... 28 days 00:00:00
  * latitude    (latitude) float64 47.0 46.93 46.87 46.8 ... 15.13 15.07 15.0
  * longitude   (longitude) float64 261.0 261.1 261.1 ... 299.9 299.9 300.0
    valid_time  (step) datetime64[ns] dask.array<chunksize=(225,), meta=np.ndarray>
Data variables:
    swh         (step, latitude, longitude) float32 dask.array<chunksize=(225, 481, 586), meta=np.ndarray>
Attributes:
    GRIB_edition:            2
...

Returned ds from open_mfdataset:

<xarray.Dataset>
Dimensions:     (latitude: 481, longitude: 586, step: 225)
Coordinates:
    time        (step) datetime64[ns] 1979-02-01 ... 1979-03-01
  * step        (step) timedelta64[ns] 0 days 00:00:00 ... 28 days 00:00:00
  * latitude    (latitude) float64 47.0 46.93 46.87 46.8 ... 15.13 15.07 15.0
  * longitude   (longitude) float64 261.0 261.1 261.1 ... 299.9 299.9 300.0
    valid_time  (step) datetime64[ns] dask.array<chunksize=(225,), meta=np.ndarray>
Data variables:
    swh         (step, latitude, longitude) float32 dask.array<chunksize=(225, 481, 586), meta=np.ndarray>
Attributes:
    GRIB_edition:            2
...

[TomNicholas]

Hi Brian,

This line

xr.open_mfdataset(files, engine='cfgrib', preprocess=preprocess, concat_dim='step', parallel=False)

should be raising a warning for you:

"When combine='by_coords', passing a value for `concat_dim` has no "
"effect. This combination will raise an error in future. To manually "
"combine along a specific dimension you should instead specify "
"combine='nested' along with a value for `concat_dim`.",

because the default value of combine='by_coords' for this function. This also means the concatenation behaviour is probably not as you are expecting. Are you using the most recent version of xarray?

[BrianBlanton]

Hi Tom, no I do not get a warning, using xarray Version = 0.19.0, via a pip install. I also just tried 0.19.1.dev15+g8f5b4a18, and do not get a warning.

But, adding combine='nested' worked.

Thank you for the answer.

[TomNicholas]

That's weird... But I'm glad it fixed your problem!

[dcherian]

I think combine_by_coords only looks at dimension coordinates (?) . In this case, the coordinates for step are mostly duplicated. So it can't infer that you want to concat along step. I don't know why it picks the second file though. Might be worth investigating.

[BrianBlanton]

that's also helpful, @dcherian . thanks very much.