Skip to content
Draft
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
53657ca
lock
cosunae Sep 22, 2025
b8c6440
add cartopy
cosunae Sep 22, 2025
4759a30
generate raw
cosunae Sep 22, 2025
4db180e
add plot rule
cosunae Sep 22, 2025
db9d4db
fix init_time
cosunae Sep 22, 2025
7faa37b
adding plotting files
cosunae Sep 22, 2025
82e0a2c
adding plotting files
cosunae Sep 22, 2025
0b29429
generate colormaps from NCL color table files
clairemerker Sep 26, 2025
bc34ec0
split colormap configs for variables and colormap loader
clairemerker Sep 30, 2025
6cb8bc0
add tests for colormap_loader.py and colormap_defaults.py
clairemerker Sep 30, 2025
9ab55d8
revert change commited accidentally
clairemerker Sep 30, 2025
12ed14c
fix typo
clairemerker Sep 30, 2025
89b3ae8
fix imports for code and tests to fit the project's structure
clairemerker Sep 30, 2025
87596a1
adapt plotting to earthkit-plots
clairemerker Oct 2, 2025
206bc07
improve tests
clairemerker Oct 2, 2025
4c57ffd
Merge branch 'main' into plotting
clairemerker Oct 2, 2025
193cb25
Merge branch 'main' into plotting
dnerini Oct 7, 2025
a6fcea6
Refactor plot rule and adopt marimo (#54)
dnerini Oct 8, 2025
6c5188d
Merge branch 'main' into plotting
dnerini Oct 8, 2025
73544e0
Fix paths
dnerini Oct 8, 2025
aefe77b
Revert using separate folder for notebooks
dnerini Oct 8, 2025
b7642cc
Specify individual showcases dates
dnerini Oct 9, 2025
69f3dc9
Switch to GRIB and fix projection issues (#60)
OpheliaMiralles Oct 14, 2025
b109eaa
Merge branch 'main' into plotting
dnerini Oct 14, 2025
7262e32
Set path to eccodes defintions
dnerini Oct 14, 2025
77b3893
Add Switzerland region to outputs
dnerini Oct 14, 2025
8fa6c67
Batch plot jobs in larger submissions
dnerini Oct 15, 2025
98b9ac9
Merge branch 'main' into plotting
dnerini Oct 15, 2025
4dd752e
Merge branch 'main' into plotting
dnerini Oct 20, 2025
421b478
Merge branch 'main' into plotting
dnerini Oct 20, 2025
61b871e
Merge branch 'main' into plotting
dnerini Oct 21, 2025
d2be5ca
Fix pre-commits
dnerini Oct 21, 2025
9df35a6
Revert change in co1e config
dnerini Oct 21, 2025
257c748
Parse steps
dnerini Oct 21, 2025
ef00ccc
Do not hardcode leadtimes
dnerini Oct 21, 2025
7306694
Plot wind speed
dnerini Oct 21, 2025
4b56b35
Showcase M-2 model
dnerini Oct 21, 2025
e50e6b2
Fix pre-commit
dnerini Oct 21, 2025
bf07cf7
Add global outputs to interpolator showcase
dnerini Oct 22, 2025
a20ae75
Fix wrong inference config...
dnerini Oct 22, 2025
d018591
Plot meteograms
dnerini Oct 22, 2025
d6fd3e3
Merge branch 'main' into plot-time-series
dnerini Nov 7, 2025
b841d19
Fix merge commit
dnerini Nov 7, 2025
dd36d53
Fix path to ICON definitions
dnerini Nov 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions config/showcase.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# yaml-language-server: $schema=../workflow/tools/config.schema.json
description: |
Basic config for showcasing M-1 forecaster.

dates:
- 2020-02-03T00:00 # Storm Petra
- 2020-02-07T00:00 # Storm Sabine
- 2020-10-01T00:00 # Storm Brigitte

runs:
- interpolator:
mlflow_id: 8d1e0410ca7d4f74b368b3079878259a
label: M-2 interpolator
steps: 0/120/1
config: resources/inference/configs/interpolator_stretched_with_global.yaml
forecaster:
mlflow_id: d0846032fc7248a58b089cbe8fa4c511
config: resources/inference/configs/forecaster_with_global.yaml
steps: 0/120/6
extra_dependencies:
- git+https://github.com/ecmwf/anemoi-inference@14189907b4f4e3b204b7994f828831b8aa51e9b6
- torch-geometric==2.6.1
- anemoi-graphs==0.5.2

baselines:
- baseline:
baseline_id: COSMO-E
label: COSMO-E
root: /store_new/mch/msopr/ml/COSMO-E
steps: 0/120/6

analysis:
label: COSMO KENDA
analysis_zarr: /scratch/mch/fzanetta/data/anemoi/datasets/mch-co2-an-archive-0p02-2015-2020-1h-v3-pl13.zarr

locations:
output_root: output/
mlflow_uri:
- https://servicedepl.meteoswiss.ch/mlstore
- https://mlflow.ecmwf.int

profile:
executor: slurm
global_resources:
gpus: 15
default_resources:
slurm_partition: "postproc"
cpus_per_task: 1
mem_mb_per_cpu: 1800
runtime: "1h"
gpus: 0
jobs: 50
batch_rules:
plot_forecast_frame: 32
3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ dependencies = [
"pydantic>=2.11.7",
"toml>=0.10.2",
"netcdf4>=1.7.2",
"cartopy",
"earthkit-plots",
"marimo>=0.16.5",
]

[project.optional-dependencies]
Expand Down
25 changes: 13 additions & 12 deletions resources/inference/configs/forecaster.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,18 @@ allow_nans: true
output:
tee:
outputs:
- extract_lam:
output:
assign_mask:
mask: "source0/trimedge_mask"
output:
grib:
path: grib/{dateTime}_{step:03}.grib
encoding:
typeOfGeneratingProcess: 2
templates:
samples: _resources/templates_index_cosmo.yaml
- printer
- extract_lam:
output:
assign_mask:
mask: "source0/trimedge_mask"
output:
grib:
path: grib/{dateTime}_{step:03}.grib
encoding:
typeOfGeneratingProcess: 2
templates:
samples: _resources/templates_index_cosmo.yaml
- printer
- raw: raw/

write_initial_state: true
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
input:
test:
use_original_paths: true

allow_nans: true

output:
tee:
outputs:
- extract_lam:
output:
grib:
path: grib/{dateTime}_{step:03}.grib
encoding:
typeOfGeneratingProcess: 2
templates:
samples: _resources/templates_index_cosmo.yaml
- grib:
path: grib/ifs-{dateTime}_{step:03}.grib
encoding:
typeOfGeneratingProcess: 2
templates:
samples: _resources/templates_index_ifs.yaml
post_processors:
- extract_slice: [919619, -1]
- assign_mask: "global/cutout_mask"

write_initial_state: true
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
runner: time_interpolator
include_forcings: true

input:
test:
Expand Down
116 changes: 116 additions & 0 deletions resources/inference/configs/interpolator_stretched_with_global.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
runner: time_interpolator

input:
cutout:
lam_0:
grib:
path: forecaster_grib/20*
pre_processors:
- extract_mask: "source0/trimedge_mask"
namer:
rules:
- - shortName: T
- t_{level}
- - shortName: U
- u_{level}
- - shortName: V
- v_{level}
- - shortName: W
- w_{level}
- - shortName: QV
- q_{level}
- - shortName: FI
- z_{level}
- - shortName: PMSL
- msl
- - shortName: FIS
- z
- - shortName: PS
- sp
- - shortName: T_2M
- 2t
- - shortName: TD_2M
- 2d
- - shortName: T_G
- skt
- - shortName: U_10M
- 10u
- - shortName: V_10M
- 10v
- - shortName: FR_LAND
- lsm
- - shortName: TOT_PREC
- tp
global:
grib:
path: forecaster_grib/ifs*
namer:
rules:
- - shortName: T
- t_{level}
- - shortName: U
- u_{level}
- - shortName: V
- v_{level}
- - shortName: W
- w_{level}
- - shortName: QV
- q_{level}
- - shortName: FI
- z_{level}
- - shortName: PMSL
- msl
- - shortName: FIS
- z
- - shortName: PS
- sp
- - shortName: T_2M
- 2t
- - shortName: TD_2M
- 2d
- - shortName: T_G
- skt
- - shortName: U_10M
- 10u
- - shortName: V_10M
- 10v
- - shortName: FR_LAND
- lsm
- - shortName: TOT_PREC
- tp

constant_forcings:
test:
use_original_paths: true

patch_metadata:
dataset:
constant_fields: [z, lsm]

output:
tee:
outputs:
- extract_lam:
output:
assign_mask:
mask: "source0/trimedge_mask"
output:
grib:
path: grib/{dateTime}_{step:03}.grib
encoding:
typeOfGeneratingProcess: 2
templates:
samples: _resources/templates_index_cosmo.yaml
- grib:
path: grib/ifs-{dateTime}_{step:03}.grib
encoding:
typeOfGeneratingProcess: 2
templates:
samples: _resources/templates_index_ifs.yaml
post_processors:
- extract_slice: [189699, -1]
- assign_mask: "global/cutout_mask"

verbosity: 1
allow_nans: true
output_frequency: "1h"
2 changes: 2 additions & 0 deletions resources/report/plotting/README
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# This is a copy of the ncl colortables developed in the cosmolib
# SRC: https://github.com/MeteoSwiss-APN/mch-ncl/tree/master/cosmolib/ct
11 changes: 11 additions & 0 deletions resources/report/plotting/RH_6lev.ct
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
; Type: COSMO Library Color Table
; Description: Color table for plotting the relative humidity in % as in NinJo
6
30 45 60 75 90 95
251 155 52
253 206 102
254 255 153
206 254 154
120 240 116
55 202 51
54 177 52
34 changes: 34 additions & 0 deletions resources/report/plotting/t2m_29lev.ct
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
; Type: COSMO Library Color Table
; Description: Color table for 2m temperature as in model browser
29
-18.0 -16.0 -14.0 -12.0 -10.0 -8.0 -6.0 -4.0 -2.0 0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 16.0 18.0 20.0 22.0 24.0 26.0 28.0 30.0 32.0 34.0 36.0 38.0
109 227 255
175 240 255
255 196 226
255 153 204
255 0 255
128 0 128
0 0 128
70 70 255
51 102 255
133 162 255
255 255 255
204 204 204
179 179 179
153 153 153
96 96 96
128 128 0
0 92 0
0 128 0
51 153 102
157 213 0
212 255 91
255 255 0
255 184 112
255 153 0
255 102 0
255 0 0
188 75 0
171 0 56
128 0 0
163 112 255
23 changes: 23 additions & 0 deletions resources/report/plotting/uv_17lev.ct
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
; Type: COSMO Library Color Table
; Description: Wind speed representation in 17 levels. This color table
; is used in the model browser to plot wind at 10m, 850, 700 and 500hPa
17
4.0 6.0 10.0 14.0 18.0 22.0 26.0 30.0 35.0 40.0 45.0 50.0 60.0 70.0 80.0 90.0 100.0
255 255 255
239 244 209
232 244 158
170 206 99
226 237 22
255 237 0
255 237 130
244 209 127
237 165 73
229 140 61
219 124 61
239 7 61
232 86 163
155 112 168
99 112 247
127 150 255
142 178 255
181 201 255
13 changes: 13 additions & 0 deletions src/evalml/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,10 @@ class Profile(BaseModel):
global_resources: GlobalResources
default_resources: DefaultResources
jobs: int = Field(..., ge=1, description="Maximum number of parallel jobs.")
batch_rules: Dict[str, int] = Field(
default_factory=dict,
description="Define batches of the same rule that shall be executed within one job submission.",
)

def parsable(self) -> Dict[str, str]:
"""Convert the profile to a dictionary of command-line arguments."""
Expand All @@ -256,6 +260,15 @@ def parsable(self) -> Dict[str, str]:
out += ["--resources"] + self.global_resources.parsable()
out += ["--default-resources"] + self.default_resources.parsable()
out += ["--jobs", str(self.jobs)]

# Add rule grouping options if specified
if self.batch_rules:
# Groups: rule=rule
groups = [f"{rule}={rule}" for rule in self.batch_rules.keys()]
# Group components: rule=<n>
components = [f"{rule}={n}" for rule, n in self.batch_rules.items()]
out += ["--groups"] + groups
out += ["--group-components"] + components
return out


Expand Down
Loading