DWD Mosmix

[cloneofghosts]

I had suggested this before but it never gained traction so am suggesting it again. The data is available for certain stations similar to #10 but includes a 240h forecast so might be useful? The data is in KML format so not sure if that makes it harder/easier to add.

More information is available here https://www.dwd.de/EN/ourservices/met_application_mosmix/met_application_mosmix.html;jsessionid=B502689E741CA864089DA8955635E33B.live21064. I can also add this to #424 if you think this might be worth investigating. @alexander0042

[cloneofghosts]

@alexander0042 Not sure if Mosmix data is something you'd be interested in adding but was curious and asked Gemini and can share if it'd be useful? I have it setup where it parses through the KML file and then stores it as a zarr file like you've done with other models.

I know you prefer to write your own code rather than use AI code but it might be a good starting point if it'd be something you'd like to add.

[alexander0042]

I'm not surprised AI didn't default to the var/time/x/y 4D array approach- it's a little bit unusual, but really optimizes retrieval times, which is why I like it

[cloneofghosts]

1. Sure you can use geopandas. I just asked how to read in data from DWD MOSMIX and store it as a zarr file.
2. Each station is a point. In the KML file there is this which is attached to each station kml:Pointkml:coordinates18.92,69.68,10.0</kml:coordinates></kml:Point>.
3. There is code in the branch which finds the nearest station and if too far away then it doesn't return anything. I have it set to 10km but you can make it smaller if needed.
4. Sure that makes sense. The code only gets the MOSMIX_S data which has a subset of the values. There's MOSMIX_L which has more variables but is run less often.
5. Sure you can do whatever you need to in order to get it working with your current systems. This was more of a proof of concept to see if it'd be something worth adding.

EDIT: Doesn't seem to be as good this afternoon with the main issue being the temperature is a lot lower than what ECCC is showing:

--- Time Step: 2025-06-26 14:00 UTC ---
  precipIntensity: 0.20 mm/h
  precipAccumulation: 0.02 cm
  precipType: rain
  temperature: 13.70°C
  dewPoint: 8.20°C
  pressure: 1023.00 hPa
  windSpeed: 18.50 km/h
  windBearing: 70°
  cloudCover: 96.00%
  visibility: 16.09 km
------------------------------

[cloneofghosts]

@alexander0042 I asked Gemini to update the code to better match the other ingest scripts and what you mentioned in your comment. It's still using the pyKML library and no idea if it'd work but it's worth a shot?

The big issue is that there's a ton of data 6000+ stations, 240h of data and 40 variables. We're saving a subset of the variables and hopefully if there are any stations that have the same coordinates as another.

I also have it save a metadata file which contains all the station metadata so you could use it to calculate the nearest station flag if you want.

[alexander0042]

TBH, that second sentence is 60% of what's hard about this API. These model output files are giant and a pain to work with, but means it's real rewarding when it works! From a first glance, gemini did an amazing job (it thought about using the same index in the 4d array between models, which would have been much smarter than what I did).

I don't have time to test this this week, but I'll give it a go early next week! The main changes I see (mostly writing this as notes for me):

1. I don't know if the pandas to xarray step helps, would rather go pandas directly to zarr
2. GFS grid should be hard coded
3. The KNN search works, but since it's regular lat long grid there's an easier way to do it with round
4. Rechunk as last step

[cloneofghosts]

I don't have time to test this this week, but I'll give it a go early next week!

Sure, that's fine. It's not particularly urgent so take the time you need to get it working. You'll probably want to remove the R101 step as it's not available in the MOSMIX_S data and only in the MOSMIX_L data (or I can ask for it to be removed)

[alexander0042]

Re your comment in the other issue- yes, totally agree that we move non CONUS to DWD. It's higher resolution and pretty highly regarded from what I can tell, so lots to like where it's available. Within CONUS, I think keep it as the priority 3 option (NBM > HRRR > DWD > GFS), but since there aern't a ton of points, I don't think it'll matter too much.

I want to add ECMWF (and maybe HRDPS/ GDPS- the Canadian models and/or Graphcast) at the same time as DWD (it's easier to just write the code once, and they're all in Herbie), so thinking it through as well. Easiest option would just be used it before GFS, since it's higher resolution, but I don't know if I love that. One option would be to implement some sort of spatial filter, and prioritize different model depending on the location- something along the lines of:

• CONUS: NBM > HRRR > DWD > GFS > GDPS > ECMWF;
• Canada: HRDPS > NBM > HRRR > DWD > GDPS > GFS > ECMWF;
• Else: DWD > ECMWF > GFS > GDPS;

Then we could add a queryparam like ?priority=NBM,HRRR.DWD.GFS,GDPS,ECMWF,Graph, or if someone wanted to try out graphcast for example ?priority=Graph.

Writing it all out, I should finish that AI code refactor first to clean up the chaos that is that script (started here, but I need to discard that and try again with recent changes). Then change the code from something like:

    # Intensity
    # NBM
    prcipIntensityHour = np.full((len(hour_array_grib), 3), np.nan)
    if "nbm" in sourceList:
        prcipIntensityHour[:, 0] = NBM_Merged[:, 13]
    # HRRR
    if ("hrrr_0-18" in sourceList) and ("hrrr_18-48" in sourceList):
        prcipIntensityHour[:, 1] = HRRR_Merged[:, 9] * 3600
    # GEFS
    if "gefs" in sourceList:
        prcipIntensityHour[:, 2] = GEFS_Merged[:, 2]
    else:  # GFS Fallback
        prcipIntensityHour[:, 2] = GFS_Merged[:, 10] * 3600

    # Take first non-NaN value
    InterPhour[:, 2] = (
        np.choose(np.argmin(np.isnan(prcipIntensityHour), axis=1), prcipIntensityHour.T)
        * prepIntensityUnit
    )

To a loop where it goes through the _MERGED variables in a priority order we can pick at the beginning depending on location, or let the user supply it via a query. Probably via pulling out that sort of block and replacing it with a function like this:

import numpy as np

def select_first_valid_value(source_data: dict, priority_order: list, column_mapping: dict, shape: tuple, scale=1.0):
    """
    Picks the first non-NaN value across sources according to the priority order.

    Parameters:
        source_data (dict): Maps source names to arrays (e.g., {'nbm': NBM_Merged}).
        priority_order (list): Ordered list of sources to prioritize.
        column_mapping (dict): Maps source names to the column index to extract.
        shape (tuple): Shape of the output array, usually (n,).
        scale (float): Optional scalar multiplier for the result.

    Returns:
        np.ndarray: Array of selected values, scaled and NaN-aware.
    """
    data_matrix = np.full((shape[0], len(priority_order)), np.nan)

    for i, source in enumerate(priority_order):
        if source in source_data and source in column_mapping:
            data_matrix[:, i] = source_data[source][:, column_mapping[source]]

    first_valid = np.choose(np.argmin(np.isnan(data_matrix), axis=1), data_matrix.T)
    return first_valid * scale

and

source_data = {
    "nbm": NBM_Merged,
    "hrrr": HRRR_Merged if ("hrrr_0-18" in sourceList and "hrrr_18-48" in sourceList) else None,
    "gefs": GEFS_Merged if "gefs" in sourceList else GFS_Merged
}

column_mapping = {
    "nbm": 13,
    "hrrr": 9,
    "gefs": 2 if "gefs" in sourceList else 10
}

priority = ["nbm", "hrrr", "gefs"]

InterPhour[:, 2] = select_first_valid_value(
    source_data,
    priority_order=priority,
    column_mapping=column_mapping,
    shape=(len(hour_array_grib),),
    scale=prepIntensityUnit
)

Although hopefully I can fix the column mappings systemically first

[cloneofghosts]

DWD does have gridded models available (see #424) but this is the only one which is updated every hour. Seems to perform the best in Europe as it seems to be pretty hit or miss in NA.

The variables available in MOSMIX_S are:

PPPP - Surface pressure, reduced
TX - Maximum temperature - within the last 12 hours
TTT - Temperature 2m above surfac
Td - Dewpoint 2m above surface
TN - Minimum temperature - within the last 12 hours
T5cm - Temperature 5cm above surface
DD - Wind direction
FF - Wind speed
FX1 - Maximum wind gust within the last hour
FX3
FXh
FXh25 - Probability of wind gusts >= 25kn within the last 12 hours
FXh40
FXh55
N - Total cloud cover
Neff - Effective cloud cover
Nh - High cloud cover (>7 km)
Nm - Midlevel cloud cover (2-7 km)
Nl - Low cloud cover (lower than 2 km)
N05 - Cloud cover below 500 ft.
VV - Visibility
wwM - Probability for fog within the last hour
wwM6
wwMh
ww - Significant Weather
W1W2 - Past weather during the last 6 hours
RR1c - Total precipitation during the last hour consistent with significant weathe
RRS1c - Snow-Rain-Equivalent during the last hour
RR3c
RRS3c
R602 - Probability of precipitation > 0.2mm during the last 6 hours
R650 - Probability of precipitation > 5.0mm during the last 6 hours
Rh00 - Probability of precipitation > 0.0mm during the last 12 hours
Rh02
Rh10
Rh50
Rd02
Rd50
Rad1h - Global Irradiance
SunD1 - Sunshine duration during the last Hour

I don't really see what else from that list to add but if there's anything to add I can add it.

[cloneofghosts]

I want to add ECMWF (and maybe HRDPS/ GDPS- the Canadian models and/or Graphcast) at the same time as DWD (it's easier to just write the code once, and they're all in Herbie), so thinking it through as well. Easiest option would just be used it before GFS, since it's higher resolution, but I don't know if I love that. One option would be to implement some sort of spatial filter, and prioritize different model depending on the location- something along the lines of:

• CONUS: NBM > HRRR > DWD > GFS > GDPS > ECMWF;
• Canada: HRDPS > NBM > HRRR > DWD > GDPS > GFS > ECMWF;
• Else: DWD > ECMWF > GFS > GDPS;

Then we could add a queryparam like ?priority=NBM,HRRR.DWD.GFS,GDPS,ECMWF,Graph, or if someone wanted to try out graphcast for example ?priority=Graph.

@alexander0042 Sorry meant to reply to this way earlier. I would think having different model priorities per location sounds interesting but for currently and minutely data it would make sense to use higher resolution and more frequently updating models over other ones.

So HRRR and NBM would be used (where available) in the US/Canada whereas for hourly Canada would prioritize HRDPS over NBM and HRRR. HRRR and NBM update more frequently so would better suit the currently and minutely blocks.

DWD is a tricky one to place in the list since it's only available around where stations are located and not every one has the same amount of data. From my testing it seems to be hit or miss in Canada (it says for 2pm that the temperature is supposed to be 24C in Ottawa which is very not true) but is better in other locations.

For testing purposes you could always set it so its used as the currently and minutely data after its integrated to see how good the data is and adjust from there? Would be interesting to compare it to the actual stations to see how close it is.

[cloneofghosts]

@alexander0042 I got AI to create a (sort of) updated version of https://github.com/krestaino/weather-vue which I published on GH pages here: https://cloneofghosts.github.io/skycast-app/

It's using the BrightSky API which uses MOSMIX_S/MOSMIX_L data so could be useful to compare and see how it performs? (feels like and RH are calculated if it doesn't exist)

[alexander0042]

Cool! Fun fact- my first idea for a name was BrihtSky, but I was worried it was too close, so picked Pirate Weather instead! And thanks for thinking of weather-vue. I never liked working with it since I didn't want to learn Vue, but if AI can figure it out, it really isn't all that complex of an app.

[cloneofghosts]

It basically turned it into more of a react app which is probably easier to handle than the weather-vue app from before.

I mainly asked for this because trying to compare this data to ECCC or other sites is hard because the BrightSky demo app is hard to find the correct hour. For the most part it seems to be pretty similar to the data other sites give so probably still worth adding as is probably better than the GFS data.