Skip to content

Commit 934028a

Browse files
pulkkinsspulkkindnerini
authored
Ide nowcasting (#221)
* Initial commit * Docstring update * Module name change * Update module docstring * Docstring additions * Set interpolation order to 1 * Change interpolation order back to 0 * Add references to IDE-based nowcasting methods * Implement utility methods * Implement computation of convolution kernel * Implement separate methods for isotropic and anisotropic kernels * Implement optimization of convolution parameters * Add the num_timesteps argument * Implement computation of window weights * Docstring revision * Add missing imports * Complete the input arguments and docstring * New version of _optimize_convol_params * New version of AR parameter optimization * Implement iteration of the AR model * Update the description of scikit-image * Add LINDA reference * Add LINDA to the nowcast interface * First version for deterministic nowcasts * Several fixes * Add more printing and measurement of computation time * Measure and return initialization and forecast loop times * Fix estimation of ARI(2,1) parameters * Remove unnecessary line * Fix estimation of isotropic kernels * Sort methods in alphabetical order * Implement separate methods for initialization and computation of deterministic nowcasts * Preparatory work for ensemble nowcasts * Implement perturbation generator * Add perturbations to deterministic nowcasts * Implement ensemble nowcasts * Implement seeding of the perturbation generator * Add TODO comment * Remove unimplemented option * Arrange methods in alphabetical order * Simplify convolution kernel and ACF computation * Numerical integration adjustments * Use exponential ACF instead of Gaussian * Raise exception if non-integer timesteps are given * Simplify parameter selection by using only one localization window radius for the deterministic model * Add arguments and docstrings needed for velocity perturbations * Minor refactoring and docstring revisions * Add default values to docstring * Add more printing * Remove unnecessary arguments in _linda_init * Implement advection field perturbation * Refactor perturbation generators * Allow input fields with odd dimensions * Remove extra output argument * Add default ARI model order to docstring * Raise exception if no features are detected * Add tests for LINDA nowcasts * Make thresholds for perturbator estimation user-configurable * Remove TODO comments * Disable printing of warnings * Add linda module * Extend testing * Limit number of realizations * Upscale mm/h; log transform as an option * Black formatting * Adapt max crps for smaller ens size * Specify in the docstring that only integer timesteps are accepted * Remove unnecessary argument * Use None instead of empty dict argument * Disable test for fractional time steps * Use f-strings instead of format * Use lowercase variable names * Simplify summation loop * Use enumerate to simplify for loop * Disable velocity perturbations by default * Replace empty dictionary argument with None * Simplify syntax of for loop * Write all function documentation into docstrings * Minor fixes Comments, unused variables, and so on * Implement measurement of coomputation time * Use smaller domain for tests * Black formatting * Fix test * Add note about the recommented number of features * Remove unused arguments * Test parallel computation with Dask * Implement separate functions for initialization of the deterministic nowcast and the perturbation generator * Docstring addition * Add recommendation about using multiple parallel workers * Use threads scheduler instead of multiprocessing * Format with black * Reduce the default number of ensemble members to 10 * Add option to use multiprocessing * Add max_num_features argument * First version of LINDA examples * Change variable name * Fix incorrect dictionary names * Disable warnings * Remove blank space before colons in docstrings * Plot S-PROG nowcast for comparison * Add discussion about the LINDA/S-PROG comparison * Add comparison with STEPS * Adjust figure size * Remove unused import * Adjust title text * Change file name * Remove ticks Co-authored-by: Seppo Pulkkinen <seppo.pulkkinen@fmi.fi> Co-authored-by: ned <daniele.nerini@meteoswiss.ch>
1 parent 3cceb10 commit 934028a

File tree

9 files changed

+1822
-17
lines changed

9 files changed

+1822
-17
lines changed

doc/source/pysteps_reference/nowcasts.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Implementation of deterministic and ensemble nowcasting methods.
88
.. automodule:: pysteps.nowcasts.interface
99
.. automodule:: pysteps.nowcasts.anvil
1010
.. automodule:: pysteps.nowcasts.extrapolation
11+
.. automodule:: pysteps.nowcasts.linda
1112
.. automodule:: pysteps.nowcasts.lagrangian_probability
1213
.. automodule:: pysteps.nowcasts.sprog
1314
.. automodule:: pysteps.nowcasts.sseps

doc/source/references.bib

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,16 @@ @ARTICLE{FNPC2020
7373
DOI = "10.3390/atmos11030267"
7474
}
7575

76+
@ARTICLE{FW2005,
77+
AUTHOR = "N. I. Fox and C. K. Wikle",
78+
TITLE = "A Bayesian Quantitative Precipitation Nowcast Scheme",
79+
JOURNAL = "Weather and Forecasting",
80+
VOLUME = 20,
81+
NUMBER = 3,
82+
PAGES = "264--275",
83+
YEAR = 2005
84+
}
85+
7686
@ARTICLE{GZ2002,
7787
AUTHOR = "U. Germann and I. Zawadzki",
7888
TITLE = "Scale-Dependence of the Predictability of Precipitation from Continental Radar Images. {P}art {I}: Description of the Methodology",
@@ -157,6 +167,14 @@ @ARTICLE{PCLH2020
157167
YEAR = 2020
158168
}
159169

170+
@ARTICLE{PCN2021,
171+
AUTHOR = "S. Pulkkinen and V. Chandrasekar and T. Niemi",
172+
TITLE = "Lagrangian Integro-Difference Equation Model for Precipitation Nowcasting",
173+
JOURNAL = "Journal of Atmospheric and Oceanic Technology",
174+
NOTE = "submitted",
175+
YEAR = 2021
176+
}
177+
160178
@INCOLLECTION{PGPO1994,
161179
AUTHOR = "M. Proesmans and L. van Gool and E. Pauwels and A. Oosterlinck",
162180
TITLE = "Determination of optical flow and its discontinuities using non-linear diffusion",
@@ -235,6 +253,16 @@ @ARTICLE{SPN2013
235253
DOI = "10.1002/wrcr.20536"
236254
}
237255

256+
@ARTICLE{XWF2005,
257+
AUTHOR = "K. Xu and C. K Wikle and N. I. Fox",
258+
TITLE = "A Kernel-Based Spatio-Temporal Dynamical Model for Nowcasting Weather Radar Reflectivities",
259+
JOURNAL = "Journal of the American Statistical Association",
260+
VOLUME = 100,
261+
NUMBER = 472,
262+
PAGES = "1133--1144",
263+
YEAR = 2005
264+
}
265+
238266
@ARTICLE{ZR2009,
239267
AUTHOR = "P. Zacharov and D. Rezacova",
240268
TITLE = "Using the fractions skill score to assess the relationship between an ensemble {QPF} spread and skill",

doc/source/user_guide/install_pysteps.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ Other optional dependencies include:
3636
* `pywavelets <https://pywavelets.readthedocs.io/en/latest/>`_
3737
(for intensity-scale verification)
3838
* `pandas <https://pandas.pydata.org/>`_ and
39-
`scikit-image <https://scikit-image.org/>`_ (for more advanced feature tracking)
39+
`scikit-image <https://scikit-image.org/>`_ (for the DATing and LINDA nowcast methods)
4040

4141
**Important**: If you only want to use pysteps, you can continue reading below.
4242
But, if you want to contribute to pysteps or edit the package, you need to install

examples/linda_nowcasts.py

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
#!/bin/env python
2+
"""
3+
LINDA nowcasts
4+
==============
5+
6+
This example shows how to compute and plot a deterministic and ensemble LINDA
7+
nowcasts using Swiss radar data.
8+
9+
"""
10+
11+
from datetime import datetime
12+
import warnings
13+
14+
warnings.simplefilter("ignore")
15+
16+
import matplotlib.pyplot as plt
17+
18+
from pysteps import io, rcparams
19+
from pysteps.motion.lucaskanade import dense_lucaskanade
20+
from pysteps.nowcasts import linda, sprog, steps
21+
from pysteps.utils import conversion, dimension, transformation
22+
from pysteps.visualization import plot_precip_field
23+
24+
###############################################################################
25+
# Read the input rain rate fields
26+
# -------------------------------
27+
28+
date = datetime.strptime("201701311200", "%Y%m%d%H%M")
29+
data_source = "mch"
30+
31+
# Read the data source information from rcparams
32+
datasource_params = rcparams.data_sources[data_source]
33+
34+
# Find the radar files in the archive
35+
fns = io.find_by_date(
36+
date,
37+
datasource_params["root_path"],
38+
datasource_params["path_fmt"],
39+
datasource_params["fn_pattern"],
40+
datasource_params["fn_ext"],
41+
datasource_params["timestep"],
42+
num_prev_files=2,
43+
)
44+
45+
# Read the data from the archive
46+
importer = io.get_method(datasource_params["importer"], "importer")
47+
reflectivity, _, metadata = io.read_timeseries(
48+
fns, importer, **datasource_params["importer_kwargs"]
49+
)
50+
51+
# Convert reflectivity to rain rate
52+
rainrate, metadata = conversion.to_rainrate(reflectivity, metadata)
53+
54+
# Upscale data to 2 km to reduce computation time
55+
rainrate, metadata = dimension.aggregate_fields_space(rainrate, metadata, 2000)
56+
57+
# Plot the most recent rain rate field
58+
plt.figure()
59+
plot_precip_field(rainrate[-1, :, :])
60+
plt.show()
61+
62+
###############################################################################
63+
# Estimate the advection field
64+
# ----------------------------
65+
66+
# The advection field is estimated using the Lucas-Kanade optical flow
67+
advection = dense_lucaskanade(rainrate, verbose=True)
68+
69+
###############################################################################
70+
# Deterministic nowcast
71+
# ---------------------
72+
73+
# Compute 30-minute LINDA nowcast with 8 parallel workers
74+
# Restrict the number of features to 15 to reduce computation time
75+
nowcast_linda = linda.forecast(
76+
rainrate,
77+
advection,
78+
6,
79+
max_num_features=15,
80+
add_perturbations=False,
81+
num_workers=8,
82+
measure_time=True,
83+
)[0]
84+
85+
# Compute S-PROG nowcast for comparison
86+
rainrate_db, _ = transformation.dB_transform(
87+
rainrate, metadata, threshold=0.1, zerovalue=-15.0
88+
)
89+
nowcast_sprog = sprog.forecast(
90+
rainrate_db[-3:, :, :],
91+
advection,
92+
6,
93+
n_cascade_levels=6,
94+
R_thr=-10.0,
95+
)
96+
97+
# Convert reflectivity nowcast to rain rate
98+
nowcast_sprog = transformation.dB_transform(
99+
nowcast_sprog, threshold=-10.0, inverse=True
100+
)[0]
101+
102+
# Plot the nowcasts
103+
fig = plt.figure(figsize=(9, 4))
104+
ax = fig.add_subplot(1, 2, 1)
105+
plot_precip_field(
106+
nowcast_linda[-1, :, :],
107+
title="LINDA (+ 30 min)",
108+
)
109+
110+
ax = fig.add_subplot(1, 2, 2)
111+
plot_precip_field(
112+
nowcast_sprog[-1, :, :],
113+
title="S-PROG (+ 30 min)",
114+
)
115+
116+
plt.show()
117+
118+
###############################################################################
119+
# The above figure shows that the filtering scheme implemented in LINDA preserves
120+
# small-scale and band-shaped features better than S-PROG. This is because the
121+
# former uses a localized elliptical convolution kernel instead of the
122+
# cascade-based autoregressive process, where the parameters are estimated over
123+
# the whole domain.
124+
125+
###############################################################################
126+
# Probabilistic nowcast
127+
# ---------------------
128+
129+
# Compute 30-minute LINDA nowcast ensemble with 40 members and 8 parallel workers
130+
nowcast_linda = linda.forecast(
131+
rainrate,
132+
advection,
133+
6,
134+
max_num_features=15,
135+
add_perturbations=True,
136+
num_ens_members=40,
137+
num_workers=8,
138+
measure_time=True,
139+
)[0]
140+
141+
# Compute 40-member STEPS nowcast for comparison
142+
nowcast_steps = steps.forecast(
143+
rainrate_db[-3:, :, :],
144+
advection,
145+
6,
146+
40,
147+
n_cascade_levels=6,
148+
R_thr=-10.0,
149+
mask_method="incremental",
150+
kmperpixel=2.0,
151+
timestep=datasource_params["timestep"],
152+
vel_pert_method=None,
153+
)
154+
155+
# Convert reflectivity nowcast to rain rate
156+
nowcast_steps = transformation.dB_transform(
157+
nowcast_steps, threshold=-10.0, inverse=True
158+
)[0]
159+
160+
# Plot two ensemble members of both nowcasts
161+
fig = plt.figure()
162+
for i in range(2):
163+
ax = fig.add_subplot(2, 2, i + 1)
164+
ax = plot_precip_field(
165+
nowcast_linda[i, -1, :, :], geodata=metadata, colorbar=False, axis="off"
166+
)
167+
ax.set_title(f"LINDA Member {i+1}")
168+
169+
for i in range(2):
170+
ax = fig.add_subplot(2, 2, 3 + i)
171+
ax = plot_precip_field(
172+
nowcast_steps[i, -1, :, :], geodata=metadata, colorbar=False, axis="off"
173+
)
174+
ax.set_title(f"STEPS Member {i+1}")
175+
176+
###############################################################################
177+
# The above figure shows the main difference between LINDA and STEPS. In
178+
# addition to the convolution kernel, another improvement in LINDA is a
179+
# localized perturbation generator using the short-space Fourier transform
180+
# (SSFT) and a spatially variable marginal distribution. As a result, the
181+
# LINDA ensemble members preserve the anisotropic and small-scale structures
182+
# considerably better than STEPS.
183+
184+
plt.tight_layout()
185+
plt.show()

pysteps/nowcasts/interface.py

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,16 +32,17 @@
3232
"""
3333

3434
from pysteps.extrapolation.interface import eulerian_persistence
35-
from pysteps.nowcasts import anvil, sprog, steps, sseps, extrapolation
35+
from pysteps.nowcasts import anvil, extrapolation, linda, sprog, steps, sseps
3636
from pysteps.nowcasts import lagrangian_probability
3737

3838
_nowcast_methods = dict()
3939
_nowcast_methods["anvil"] = anvil.forecast
4040
_nowcast_methods["eulerian"] = eulerian_persistence
4141
_nowcast_methods["extrapolation"] = extrapolation.forecast
4242
_nowcast_methods["lagrangian"] = extrapolation.forecast
43-
_nowcast_methods["probability"] = lagrangian_probability.forecast
4443
_nowcast_methods["lagrangian_probability"] = lagrangian_probability.forecast
44+
_nowcast_methods["linda"] = linda.forecast
45+
_nowcast_methods["probability"] = lagrangian_probability.forecast
4546
_nowcast_methods["sprog"] = sprog.forecast
4647
_nowcast_methods["sseps"] = sseps.forecast
4748
_nowcast_methods["steps"] = steps.forecast
@@ -68,8 +69,10 @@ def get_method(name):
6869
| lagrangian or | this approach extrapolates the last observation |
6970
| extrapolation | using the motion field (Lagrangian persistence) |
7071
+-----------------+-------------------------------------------------------+
72+
| linda | the LINDA method developed in :cite:`PCN2021` |
73+
+-----------------+-------------------------------------------------------+
7174
| lagrangian_\ | this approach computes local lagrangian probability |
72-
| probability | forecasts of threshold exceedences. |
75+
| probability | forecasts of threshold exceedences |
7376
+-----------------+-------------------------------------------------------+
7477
| sprog | the S-PROG method described in :cite:`Seed2003` |
7578
+-----------------+-------------------------------------------------------+
@@ -78,7 +81,7 @@ def get_method(name):
7881
| | |
7982
+-----------------+-------------------------------------------------------+
8083
| sseps | short-space ensemble prediction system (SSEPS). |
81-
| | Essentially, this is a localization of STEPS. |
84+
| | Essentially, this is a localization of STEPS |
8285
+-----------------+-------------------------------------------------------+
8386
"""
8487
if isinstance(name, str):

0 commit comments

Comments
 (0)