Challenge 22 - Physical consistency in multi-variate Machine Learning applications for the Earth System

[DiegoDomenici]

Challenge 22 - Physical consistency in multi-variate Machine Learning applications for the Earth System

Stream 2 - Machine Learning for Earth Sciences Applications

Goal

Develop an evaluation framework for inter-variable physical consistency.

Mentors

Michael Langguth, Erik Pavel, Sindhu Vasireddy, Ankit Patnala, Savvas Melidonis (Jülich)
Christopher Goddard (ECMWF)

Skills Required

• Experience with Python programming
• Large-scale data analysis
• Statistical analysis
• Background in Meteorology or related fields
• Basic experience in Machine Learning desirable

---

Description

Over the past few years, Machine Learning has been applied to meteorological applications such as forecasting and statistical downscaling with growing success. These data-driven methods nowadays rival or surpass the performance of state-of-the-art numerical weather prediction (NWP) models or classical statistical techniques. While first studies suggest that ML-based forecasting models can learn the underlying physics of atmospheric dynamics (Hakim et al., 2024 [1]), the extent to which ML models ensure physical consistency in multi-variate applications remains unclear.

In particular, statistical downscaling with deep neural networks nowadays outperform classical approaches such as ECPoint. Recent advancement such as the CorrDiff downscaling model (Mardani et al., 2023 [2]) show promise in multi-variate downscaling applications. The results indicate that physical consistency is warranted in the downscaled meteorological fields, even though an image-to-image approach is applied. However, a systematic analysis of inter-variable physical consistency is still outstanding. This evaluation is crucial, particularly when these ML products aim to emulate high-resolution physics-based NWP models.

The goal of this challenge is to develop an evaluation framework that systematically assesses inter-variable physical consistency in ML-based applications. In scope of this challenge, inference data from the multi-variate CorrDiff downscaling model will be analysed. Despite the focus on statistical downscaling, the framework should be versatile enough to apply to other ML-based meteorological products.

What we offer

• Advanced Python skills for dedicated packages such as numpy, xarray and dask
• Access to the Jülich HPC system for data analysis at scale
• Advancing the development of multi-variate downscaling applications

Ideas for the challenge implementation

• Identification of suitable evaluation metrics
• Research and implementation of dedicated metrics for evaluating cross-variable consistency, as well as metrics for spatial and temporal coherence (e.g. extending the work by Borchet et al., 2023 [3])
• Visualization and diagnostic tools
• Create a framework that provides an overview on physical consistency. This can either be a simple Python-script with suitable visualizations such as in WeatherBench 2 (Rasp et al., 2023 [4]) or a dashboard.
• Validation on a downscaling product
• The framework will be tested on the downscaling product of the CorrDiff model.
• Case studies for further insight
• A set of case studies for key meteorological phenomena will be further evaluated. The selection of these phenomena, e.g. cold fronts, convective events or patterns due to land-sea breeze, as well as their evaluation criteria shall be protocoled

[1] Hakim, Gregory J., and Sanjit Masanam. "Dynamical tests of a deep-learning weather prediction model." Artificial Intelligence for the Earth Systems (2024).
[2] Mardani, Morteza, et al. "Residual corrective diffusion modeling for km-scale atmospheric downscaling." URL https://arxiv. org/abs/2309.15214 (2023).
[3] Brochet, Clément, et al. "Multivariate Emulation of Kilometer-Scale Numerical Weather Predictions with Generative Adversarial Networks: A Proof of Concept." Artificial Intelligence for the Earth Systems 2.4 (2023): 230006.
[4] Rasp, Stephan, et al. "WeatherBench 2: A benchmark for the next generation of data‐driven global weather models." Journal of Advances in Modeling Earth Systems 16.6 (2024): e2023MS004019.

---

Evaluation Criteria

• Feasibility
• Transferability
• Matching requirements

[surbhigoel77]

Hi Team. I am interested in applying for this project. Do you expect participants to have good domain knowledge as well, as the project is about evaluating the physical consistency of predictions?

[mlangguth89]

Hi @surbhigoel77
Thanks for your interest in our challenge! 😊

Yes, having meteorological knowledge is beneficial, as we aim to explore physical consistency not only through a statistical evaluation framework but also via case studies. Use cases focusing on specific weather phenomena (e.g., the passage of a cold front) provide an intuitive way to assess consistency and can reveal strengths and weaknesses that might not be apparent from metric-based evaluations alone.

While our team of mentors includes meteorologists, it would be helpful for applying teams to have meteorological expertise as well. This will ensure that the challenge objectives can be effectively addressed, particularly when interpreting meteorological scenarios and evaluating physical consistency.