om metadata

[kikocorreoso]

Hi,

I have been playing with omfiles (it is great, thanks!!!) and the basic functionality is really useful.

I was wondering if there is/will be an easy way to better autodiscover information about the data itself, the metadata, names of the variables, dimensions, etc.

If this is not yet ready, how can I help?

[terraputix]

Hi, first, thanks for your feedback!

Opening an existing .om-file with xarray should already partially deliver this functionality, e.g. compare this test case.
I think some of the functionality used in xarray_backend.py should potentially be improved and generalized so we can reuse it elsewhere. Currently, the method get_flat_variable_metadata is used to traverse all the children and get a directory-like mapping to the offset and size in the file. Though, I doubt this is an optimal design and I might need to change it later, at least in order to be evaluated more lazily.

As a side note: the files currently produced by open-meteo have literally no metadata associated with them. Currently, they only contain one single variable without a name assigned, compare here. Thus, if you are using files created by open-meteo there is no real metadata to discover at the moment.

I am currently also investigating how the open-meteo weather-data catalogue on S3 could be accessed in a more zarr-like manner with VirtualiZarr and easily discovering metadata will definitely be required for this!

Maybe you could add some information what your exact use case for omfiles is? I am curios about that and it would help me to prioritize future developments.

[kikocorreoso]

Thanks for the explanation.

I see each file is just for a variable. The code I have been using for my tests is taken from the tests in omfiles (omfiles.OmFilePyReader). This is using numpy under the hood IIRC. So, it seems, there are some other ways to get more complete information. I see the following in the tests:

reader = om.OmFilePyReader(temp_file)
backend_array = om_xarray.OmBackendArray(reader=reader)

Sorry for not having a look to this. I will check it later.

One of my main usages would be for lat-lon plots on a specific timestamp. The discoverability of the data would be useful for different sources (timestamps with 1h or 3h, lat-lon from a global or a LAM model, etc).

[terraputix]

I see each file is just for a variable.

It does not necessarily have to be that way, but that is how open-meteo is currently using this file format. The file format itself could really be used to store an arbitrary number of variables.

This is using numpy under the hood IIRC.

The __getitem__ method is using numpy under the hood, but there are some (limited) meta-information available directly on the OmFilePyReader such as shape, dtype and name.

Generally, I totally agree with you! It should be more straight-forward to find out necessary meta-information about data when opening a specific file.

For your use case, I set up a small example script in the following PR: https://github.com/terraputix/omfilesrspy/pull/24/files.
I think there you can see how latitudes, longitudes and times for a specific domain can be derived at the moment. It definitely needs too much expert knowledge (and potentially the current implementation is also not 100% free of any bugs) to be usable for a broader audience. Anyway feel free to toy around with it and get back to me with potential questions/fixes/remarks.

The extensive knowledge currently required to read the publicly available om-files from the S3-bucket, is something which I might be able to solve with kerchunk/virtualizarr and I am still looking into that!

cc: @patrick-zippenfenig

[kikocorreoso]

Thanks for the example. Very useful to better understand how to read the files. And sorry for not coming back after a while. Too many open issues.

What I see is that, independently if you use kerchunk/virtualizarr/omfiles, you will need to add metadata in some way.

meta.json helps to find the correct chunk/time. Could/should this functionality, provided in your example, be included within omfiles lib?

On the other hand for the grid definition, as you comment, the information is mainly hardcoded in different swift files. Would it make sense to add a grid.om file in each static/ folder for each of the models? But with this I'm also wondering:

• Depending the model/variable you could have several grids, node centered, cell centered,...
• How to define the correct grid.om file when there is an update of a model (other grid, other resolution, etc).

Again, thank you very much for the example and let me know if I could help in some way.

[patrick-zippenfenig]

I would like to add most metadata information like the number of time-steps in each chunk and grid information into meta.json.

Currently, Open-Meteo uses regular grids, gaussian grids and projections like Lambert conical, Lambert Azimuth, rotated lat-on and stereographic. Each grid/projection has a couple of parameters. Although I have been working with libraries like proj4 in the past, I find it very hard to define this information in a projections string. Are there any other meta data standards for grids and projections? E.g. the eccodes library for GRIB files offers a lot of attributes like latitudeOfFirstGridPointInDegrees, Nx/Ny, LaDInDegrees, LoVInDegrees, DxInMetres, DyInMetres, Latin1InDegrees and many more. I could recreates those attributes in the meta.json.

Note: Currently the Open-Meteo database on AWS only provides continuous time-series data in chunks or yearly files. I am planning to also publish spatially oriented data to generate maps or store individual runs. Spatial data may have different resolutions in the same file like GeoTiff. This should be considered for meta data.

[kikocorreoso]

I would like to add most metadata information like the number of time-steps in each chunk and grid information into meta.json.

Currently, Open-Meteo uses regular grids, gaussian grids and projections like Lambert conical, Lambert Azimuth, rotated lat-on and stereographic. Each grid/projection has a couple of parameters. Although I have been working with libraries like proj4 in the past, I find it very hard to define this information in a projections string. Are there any other meta data standards for grids and projections? E.g. the eccodes library for GRIB files offers a lot of attributes like latitudeOfFirstGridPointInDegrees, Nx/Ny, LaDInDegrees, LoVInDegrees, DxInMetres, DyInMetres, Latin1InDegrees and many more. I could recreates those attributes in the meta.json.

I think the CF conventions are the most extended convention out there. The different projections used by the models in open-meteo are defined there. Section 5.6 is the most interesting one.

(Geo)Zarr is adhering to the CF conventions.

In the case of *.om files where everything is stored as 2D arrays the vertical and time coordinates would be interesting to be included as metadata and latitude and longitude could be included directly as an static *.om file like topography/elevation/etc of the model. In this case, a grid_mapping metadata variable/section could provide more context but it wouldn't be mandatory. I think this would be the most direct way to make *.om files easier to be used.

Note: Currently the Open-Meteo database on AWS only provides continuous time-series data in chunks or yearly files. I am planning to also publish spatially oriented data to generate maps or store individual runs. Spatial data may have different resolutions in the same file like GeoTiff. This should be considered for meta data.

Spatial-friendly indexing is pretty interesting for climatology topics.

Let me know if I can help in some way.

[terraputix]

Although I have been working with libraries like proj4 in the past, I find it very hard to define this information in a projections string.

I have been digging into this and I think I found all the relevant information needed for valid proj.4 strings for the projections currently used by open-meteo, so that should not really be a problem. There is also a document specifying quite extensively how to translate between cf-convention parameter names and proj.4 strings here.

meta.json helps to find the correct chunk/time. Could/should this functionality, provided in your example, be included within omfiles lib?

I would like to add most metadata information like the number of time-steps in each chunk and grid information into meta.json.

I agree that we should expose all the necessary meta-information in the meta.json file (including the necessary parameters for the projection-grids and number of timestamps), otherwise I would need to hardcode/duplicate this logic in the python library.

I would suggest a structure like the following (e.g. for cmc_gem_rdps):

{
  "data_end_time":1748026800,
  "last_run_availability_time":1747732915,
  "last_run_initialisation_time":1747720800,
  "last_run_modification_time":1747732915,
  "temporal_resolution_seconds":3600,
  "update_interval_seconds":21600,

  "chunk_files_dimensions": {
       "nx": 935,
       "ny": 824,
       "nt": 114
   },
  "grid_mapping_name": "stereographic",
  "grid_mapping_parameters": {
      "latitude_of_projection_origin": 90.0,
      "straight_vertical_longitude_from_pole": 249.0,
      "earth_radius": 6371229.0
  },
  "grid_bounds": {
      "lat_range": [18.14503, 45.405453],
      "lon_range": [217.10745, 349.8256]
  },
  "proj_string": "+proj=stere +lat_0=90 +lon_0=249 +k=1 +x_0=0 +y_0=0 +R=6371229 +units=m +no_defs"
}

Is there anything I am missing in that structure or anything that needs to be improved?

If you want, I could start to implement and test this approach @patrick-zippenfenig ?

[kikocorreoso]

I agree that we should expose all the necessary meta-information in the meta.json file (including the necessary parameters for the projection-grids and number of timestamps), otherwise I would need to hardcode/duplicate this logic in the python library.

I think this would be the easiest way to generalize the access, not only from python, the the datasets.

Is there anything I am missing in that structure or anything that needs to be improved?

This would be at model level but an update to a model would be treated as a new model (homogeneity, resolution changes, model updates, etc)? Or as a new version of the meta.json? I think a version field and maybe some fields to map dates to meta.json version would be interesting. Something like:

[
	{
	  "version":0,
	  "data_end_time":1748026800,
	  "last_run_availability_time":1747732915,
	  "last_run_initialisation_time":1747720800,
	  "last_run_modification_time":1747732915,
	  "temporal_resolution_seconds":3600,
	  "update_interval_seconds":21600,

	  "chunk_files_dimensions": {
		   "nx": 935,
		   "ny": 824,
		   "nt": 114
	   },
	  "grid_mapping_name": "stereographic",
	  "grid_mapping_parameters": {
		  "latitude_of_projection_origin": 90.0,
		  "straight_vertical_longitude_from_pole": 249.0,
		  "earth_radius": 6371229.0
	  },
	  "grid_bounds": {
		  "lat_range": [18.14503, 45.405453],
		  "lon_range": [217.10745, 349.8256]
	  },
	  "proj_string": "+proj=stere +lat_0=90 +lon_0=249 +k=1 +x_0=0 +y_0=0 +R=6371229 +units=m +no_defs"
	},
	{
	  "version":1,
	  ...	  
	},
]

Is this a premature optimization? Model setups change oftenly and changing something like meta.json would be more difficult.

[terraputix]

Model setups change oftenly and changing something like meta.json would be more difficult.

It is true that the NWP models are continuously worked on and refined, but fundamental things such as the model grid usually don't change very often (e.g. icon-eu in November of 2023). If they do, some logic in the application layer would be needed to merge different model grids/resolutions for things such as the historical forecast api.

I agree that we should think about this at this step and your suggestion is a potential solution! Another solution might be moving the forecasts with the new (or old) model resolution to another model key (e.g. dwd_icon_eu_pre_23112023). I honestly prefer your suggestion, but I am concerned that it will be harder to compare historical forecasts over time-periods spanning such major model enhancements.

[terraputix]

I have been digging into this and I think I found all the relevant information needed for valid proj.4 strings for the projections currently used by open-meteo, so that should not really be a problem. There is also a document specifying quite extensively how to translate between cf-convention parameter names and proj.4 strings here.

I have to correct myself here slightly. For regular grids (such as regular lat-lon grids or regular projected grids) the above mentioned approach would work. For irregular grids such as Gaussian Grids (e.g. as used in ecmwf_ifs) the above approach wouldn't work.

[patrick-zippenfenig]

If they do, some logic in the application layer would be needed to merge different model grids/resolutions for things such as the historical forecast api.

Yes, a change in the grid definition should change the name of the dataset. E.g. ecmwf_ifs025 super-seeds ecmwf_ifs04.

Is there anything I am missing in that structure or anything that needs to be improved?

Looks really good. The grid_mapping_* terms seem to work quite well with climate conventions metadata. Are the terms grid_bounds and lat_range somehow in the standard as well? I did not find any references.

[terraputix]

Are the terms grid_bounds and lat_range somehow in the standard as well? I did not find any references.

They are not and I fear there is not really a CF-conformant way to define them (at least I did not find any when parsing the CF-conventions again).

Usually, the CF-conventions would require us to store two variables with corresponding standard_names called projection_y_coordinate and projection_x_coordinate (compare example 5.7).

In a way, I guess the grid_bounds would correspond to what the eccodes library is doing with latitudeOfFirstGridPointInDegrees and similar parameters.

E.g. the eccodes library for GRIB files offers a lot of attributes like latitudeOfFirstGridPointInDegrees, Nx/Ny, LaDInDegrees, LoVInDegrees, DxInMetres, DyInMetres, Latin1InDegrees and many more. I could recreates those attributes in the meta.json.