Skip to content

Commit f014669

Browse files
cosunaeclairemerkerdneriniOpheliaMirallesjonasbhend
authored
MRB-422 Generate plots for inference output (#50)
* lock * add cartopy * generate raw * add plot rule * fix init_time * adding plotting files * adding plotting files * generate colormaps from NCL color table files * split colormap configs for variables and colormap loader * add tests for colormap_loader.py and colormap_defaults.py * revert change commited accidentally * fix typo * fix imports for code and tests to fit the project's structure * adapt plotting to earthkit-plots * improve tests * Refactor plot rule and adopt marimo (#54) * Adopt more atomic approach Also use marimo for interactive editing * Move notebook to dedicated folder * Remove original script * Add example config * Revert some wrong changes * Fix paths * Revert using separate folder for notebooks * Specify individual showcases dates * Switch to GRIB and fix projection issues (#60) * Adopt more atomic approach Also use marimo for interactive editing * Move notebook to dedicated folder * add plot rule * Revert some wrong changes * Solve bugs adopt grib * Replace by plot_forecast_frame.py * Rename * Fix number of points cosmo 1e * Remove globe because it doesn't seem right * Add possibility to set region to none for global plots * Review * Update workflow/scripts/src/plotting.py * Simplify code * Set path to eccodes defintions * Add Switzerland region to outputs * Batch plot jobs in larger submissions * Fix pre-commits * Revert change in co1e config * Parse steps * Do not hardcode leadtimes * Plot wind speed I'm assuming the default colormap is in knots * Showcase M-2 model * Fix pre-commit * Add global outputs to interpolator showcase * Fix wrong inference config... * Plot only candidate runs * Add units to comment * Revert changes to inference config * Unify projections and regions in one object * Remove debugging code... * Add europe region to outputs * Use modified colormap for wind * Reduce figure size and quality * Fix inputs to plot rule * Debugging * Add previous commit hash to interpolators * Fix artefacts in plots * Precommit * Remove unwanted change * Add lam boundaries * add stratification to satisfy pydantic * Update config/showcase.yaml Co-authored-by: Jonas Bhend <jonasbhend@users.noreply.github.com> * Fix creation of header * Add plots for tp * Fix tests * Use ifs name 'si' for wind speed instead of 'sp' * Dynamic delay for gif animations * Update workflow/rules/plot.smk Co-authored-by: Daniele Nerini <daniele.nerini@meteoswiss.ch> * run pre-commit * Adopt ICON names after fix in dea87ce * Switch to main branch of anemoi-inference * Pre-commit --------- Co-authored-by: clairemerker <34312518+clairemerker@users.noreply.github.com> Co-authored-by: Daniele Nerini <daniele.nerini@meteoswiss.ch> Co-authored-by: Daniele Nerini <daniele.nerini@gmail.com> Co-authored-by: Ophélia Miralles <opheliamiralles@gmail.com> Co-authored-by: omiralles <ophelia.miralles@meteoswiss.ch> Co-authored-by: Jonas Bhend <jonas.bhend@meteoswiss.ch> Co-authored-by: Jonas Bhend <jonasbhend@users.noreply.github.com> Co-authored-by: Francesco Zanetta <francesco.zanetta@meteoswiss.ch>
1 parent aac2702 commit f014669

26 files changed

+1567
-17
lines changed

‎config/forecasters-co1e.yaml‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,19 +4,21 @@ description: |
44
(KENDA-1) at 1km resolution.
55
66
dates:
7-
start: 2020-08-01T12:00
8-
end: 2020-08-10T00:00
7+
start: 2024-01-01T12:00
8+
end: 2024-01-10T00:00
99
frequency: 54h
1010

1111
runs:
1212
- forecaster:
1313
mlflow_id: 2174c939c8844555a52843b71219d425
1414
label: Cosmo 1km + era5 N320, finetuned on cerra checkpoint, lam resolution 11
15-
config: resources/inference/configs/forecaster_no_trimedge_fromtraining.yaml
15+
config: resources/inference/configs/forecaster_no_trimedge_with_global.yaml
1616
steps: 0/120/6
1717
inference_resources:
1818
gpu: 4
1919
tasks: 4
20+
extra_dependencies:
21+
- git+https://github.com/ecmwf/anemoi-inference.git@main
2022

2123
baselines:
2224
- baseline:

‎config/forecasters.yaml‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,18 @@ description: |
33
Evaluate skill of COSMO-E emulator (M-1 forecaster).
44
55
dates:
6-
start: 2020-01-01T12:00
7-
end: 2020-01-10T00:00
8-
frequency: 60h
6+
- 2020-02-03T00:00 # Storm Petra
7+
- 2020-02-07T00:00 # Storm Sabine
8+
- 2020-10-01T00:00 # Storm Brigitte
99

1010
runs:
1111
- forecaster:
1212
mlflow_id: d0846032fc7248a58b089cbe8fa4c511
1313
label: M-1 forecaster
1414
steps: 0/120/6
15+
config: resources/inference/configs/forecaster_with_global.yaml
16+
extra_dependencies:
17+
- git+https://github.com/ecmwf/anemoi-inference.git@main
1518

1619
baselines:
1720
- baseline:

‎config/interpolators.yaml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ runs:
1515
config: resources/inference/configs/interpolator_from_test_data_stretched.yaml
1616
forecaster: null
1717
extra_dependencies:
18-
- git+https://github.com/ecmwf/anemoi-inference@fix/cutout-preprocessors
18+
- git+https://github.com/ecmwf/anemoi-inference.git@main
1919
- torch-geometric==2.6.1
2020
- anemoi-graphs==0.5.2
2121
- interpolator:
@@ -28,14 +28,16 @@ runs:
2828
config: resources/inference/configs/forecaster_with_global.yaml
2929
steps: 0/120/6
3030
extra_dependencies:
31-
- git+https://github.com/ecmwf/anemoi-inference@fix/cutout-preprocessors
31+
- git+https://github.com/ecmwf/anemoi-inference.git@main
3232
- torch-geometric==2.6.1
3333
- anemoi-graphs==0.5.2
3434
- forecaster:
3535
mlflow_id: d0846032fc7248a58b089cbe8fa4c511
3636
label: M-1 forecaster
3737
config: resources/inference/configs/forecaster_with_global.yaml
3838
steps: 0/120/6
39+
extra_dependencies:
40+
- git+https://github.com/ecmwf/anemoi-inference.git@main
3941

4042
baselines:
4143
- baseline:

‎config/showcase.yaml‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# yaml-language-server: $schema=../workflow/tools/config.schema.json
2+
description: |
3+
Basic config for showcasing M-2 interpolator on M-1 forecaster.
4+
5+
dates:
6+
- 2020-02-03T00:00 # Storm Petra
7+
- 2020-02-07T00:00 # Storm Sabine
8+
- 2020-10-01T00:00 # Storm Brigitte
9+
10+
runs:
11+
- interpolator:
12+
mlflow_id: 8d1e0410ca7d4f74b368b3079878259a
13+
label: M-2 interpolator
14+
steps: 0/120/1
15+
config: resources/inference/configs/interpolator_stretched_with_global.yaml
16+
forecaster:
17+
mlflow_id: d0846032fc7248a58b089cbe8fa4c511
18+
config: resources/inference/configs/forecaster_with_global.yaml
19+
steps: 0/120/6
20+
extra_dependencies:
21+
- git+https://github.com/ecmwf/anemoi-inference.git@main
22+
extra_dependencies:
23+
- git+https://github.com/ecmwf/anemoi-inference.git@main
24+
- torch-geometric==2.6.1
25+
- anemoi-graphs==0.5.2
26+
27+
baselines:
28+
- baseline:
29+
baseline_id: COSMO-E
30+
label: COSMO-E
31+
root: /store_new/mch/msopr/ml/COSMO-E
32+
steps: 0/120/6
33+
34+
stratification:
35+
regions:
36+
- jura
37+
- mittelland
38+
- voralpen
39+
- alpennordhang
40+
- innerealpentaeler
41+
- alpensuedseite
42+
root: /scratch/mch/bhendj/regions/Prognoseregionen_LV95_20220517
43+
44+
analysis:
45+
label: COSMO KENDA
46+
analysis_zarr: /scratch/mch/fzanetta/data/anemoi/datasets/mch-co2-an-archive-0p02-2015-2020-6h-v3-pl13.zarr
47+
48+
locations:
49+
output_root: output/
50+
mlflow_uri:
51+
- https://servicedepl.meteoswiss.ch/mlstore
52+
- https://mlflow.ecmwf.int
53+
54+
profile:
55+
executor: slurm
56+
global_resources:
57+
gpus: 15
58+
default_resources:
59+
slurm_partition: "postproc"
60+
cpus_per_task: 1
61+
mem_mb_per_cpu: 1800
62+
runtime: "1h"
63+
gpus: 0
64+
jobs: 50
65+
batch_rules:
66+
plot_forecast_frame: 32

‎pyproject.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,9 @@ dependencies = [
2121
"shapely>=2.1.2",
2222
"cartopy>=0.25.0",
2323
"pyproj>=3.7.2",
24+
"earthkit-plots",
25+
"marimo>=0.16.5",
26+
"geopandas>=0.14.0",
2427
]
2528

2629
[project.optional-dependencies]

‎resources/inference/configs/forecaster_no_trimedge_with_global.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
input:
2+
test:
3+
use_original_paths: true
4+
5+
allow_nans: true
6+
7+
output:
8+
tee:
9+
outputs:
10+
- extract_lam:
11+
output:
12+
grib:
13+
path: grib/{dateTime}_{step:03}.grib
14+
encoding:
15+
typeOfGeneratingProcess: 2
16+
templates:
17+
samples: resources/templates_index_cosmo.yaml
18+
- grib:
19+
path: grib/ifs-{dateTime}_{step:03}.grib
20+
encoding:
21+
typeOfGeneratingProcess: 2
22+
templates:
23+
samples: resources/templates_index_ifs.yaml
24+
post_processors:
25+
- extract_mask:
26+
mask: "lam_0/cutout_mask"
27+
as_slice: true
28+
inverse: true # removes lam points
29+
- assign_mask:
30+
mask: "global/cutout_mask" # fill local/global overlapping points with nan
31+
32+
write_initial_state: true

‎resources/inference/configs/forecaster_with_global.yaml‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,12 @@ output:
2525
templates:
2626
samples: resources/templates_index_ifs.yaml
2727
post_processors:
28-
- extract_slice: [189699, -1]
29-
- assign_mask: "global/cutout_mask"
28+
- extract_mask:
29+
mask: "lam_0/cutout_mask"
30+
as_slice: true
31+
inverse: true # removes lam points
32+
- assign_mask:
33+
mask: "global/cutout_mask" # fill local/global overlapping points with nan
3034

3135
constant_forcings:
3236
test:

‎resources/inference/configs/interpolator_stretched_with_global.yaml‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
runner: time_interpolator
2+
3+
input:
4+
cutout:
5+
lam_0:
6+
grib:
7+
path: forecaster/20*
8+
pre_processors:
9+
- extract_mask: "source0/trimedge_mask"
10+
namer:
11+
rules:
12+
- - shortName: T
13+
- t_{level}
14+
- - shortName: U
15+
- u_{level}
16+
- - shortName: V
17+
- v_{level}
18+
- - shortName: W
19+
- w_{level}
20+
- - shortName: QV
21+
- q_{level}
22+
- - shortName: FI
23+
- z_{level}
24+
- - shortName: PMSL
25+
- msl
26+
- - shortName: FIS
27+
- z
28+
- - shortName: PS
29+
- sp
30+
- - shortName: T_2M
31+
- 2t
32+
- - shortName: TD_2M
33+
- 2d
34+
- - shortName: T_G
35+
- skt
36+
- - shortName: U_10M
37+
- 10u
38+
- - shortName: V_10M
39+
- 10v
40+
- - shortName: FR_LAND
41+
- lsm
42+
- - shortName: TOT_PREC
43+
- tp
44+
global:
45+
grib:
46+
path: forecaster/ifs*
47+
namer:
48+
rules:
49+
- - shortName: T
50+
- t_{level}
51+
- - shortName: U
52+
- u_{level}
53+
- - shortName: V
54+
- v_{level}
55+
- - shortName: W
56+
- w_{level}
57+
- - shortName: QV
58+
- q_{level}
59+
- - shortName: FI
60+
- z_{level}
61+
- - shortName: PMSL
62+
- msl
63+
- - shortName: FIS
64+
- z
65+
- - shortName: PS
66+
- sp
67+
- - shortName: T_2M
68+
- 2t
69+
- - shortName: TD_2M
70+
- 2d
71+
- - shortName: T_G
72+
- skt
73+
- - shortName: U_10M
74+
- 10u
75+
- - shortName: V_10M
76+
- 10v
77+
- - shortName: FR_LAND
78+
- lsm
79+
- - shortName: TOT_PREC
80+
- tp
81+
82+
constant_forcings:
83+
test:
84+
use_original_paths: true
85+
86+
patch_metadata:
87+
dataset:
88+
constant_fields: [z, lsm]
89+
90+
output:
91+
tee:
92+
outputs:
93+
- extract_lam:
94+
output:
95+
assign_mask:
96+
mask: "source0/trimedge_mask"
97+
output:
98+
grib:
99+
path: grib/{dateTime}_{step:03}.grib
100+
encoding:
101+
typeOfGeneratingProcess: 2
102+
templates:
103+
samples: resources/templates_index_cosmo.yaml
104+
- grib:
105+
path: grib/ifs-{dateTime}_{step:03}.grib
106+
encoding:
107+
typeOfGeneratingProcess: 2
108+
templates:
109+
samples: resources/templates_index_ifs.yaml
110+
post_processors:
111+
- extract_mask:
112+
mask: "lam_0/cutout_mask"
113+
as_slice: true
114+
inverse: true # removes lam points
115+
- assign_mask:
116+
mask: "global/cutout_mask" # fill local/global overlapping points with nan
117+
118+
119+
verbosity: 1
120+
allow_nans: true
121+
output_frequency: "1h"

‎resources/report/plotting/README‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# This is a copy of the ncl colortables developed in the cosmolib
2+
# SRC: https://github.com/MeteoSwiss-APN/mch-ncl/tree/master/cosmolib/ct

‎resources/report/plotting/RH_6lev.ct‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
; Type: COSMO Library Color Table
2+
; Description: Color table for plotting the relative humidity in % as in NinJo
3+
6
4+
30 45 60 75 90 95
5+
251 155 52
6+
253 206 102
7+
254 255 153
8+
206 254 154
9+
120 240 116
10+
55 202 51
11+
54 177 52

0 commit comments

Comments
 (0)