Skip to content

Commit b099c3e

Browse files
Merge pull request #2060 from CliMA/ck/vert_remapper
Extend Remapper to FiniteDifferenceSpaces, update interpolate interface
2 parents 81a8fa1 + 69c02ac commit b099c3e

File tree

10 files changed

+836
-99
lines changed

10 files changed

+836
-99
lines changed

NEWS.md

Lines changed: 24 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -4,20 +4,28 @@ ClimaCore.jl Release Notes
44
main
55
-------
66

7+
### ![][badge-✨feature/enhancement] Various improvements to `Remapper` [2060](https://github.com/CliMA/ClimaCore.jl/pull/2060)
8+
9+
The `ClimaCore.Remapping` module received two improvements. First, `Remapper` is
10+
now compatible with purely vertical `Space`s (performing a linear
11+
interpolation), making it compatible with column setups. Second, a new set of
12+
simplified interpolation functions are provided.
13+
14+
Now, interpolating a `Field` `field` is as easy as
15+
```julia
16+
import ClimaCore.Remapping: interpolate
17+
output_array = interpolate(field)
18+
```
19+
The target coordinates are automatically determined, but can also be customized.
20+
Refer to the [documentation](https://clima.github.io/ClimaCore.jl/dev/remapping/)
21+
for more information.
22+
723
v0.14.21
824
--------
925

1026
- Support for new TVD limiters were added, PR [1662]
1127
(https://github.com/CliMA/ClimaCore.jl/pull/1662).
1228

13-
- We've added new convenience constructors for spaces PR [2082](https://github.com/CliMA/ClimaCore.jl/pull/2082). Here are links to the new constructors:
14-
- [ExtrudedCubedSphereSpace]()
15-
- [CubedSphereSpace]()
16-
- [ColumnSpace]()
17-
- [Box3DSpace]()
18-
- [SliceXZSpace]()
19-
- [RectangleXYSpace]()
20-
2129
### ![][badge-🐛bugfix] Bug fixes
2230

2331
- Fixed writing/reading purely vertical spaces. PR [2102](https://github.com/CliMA/ClimaCore.jl/pull/2102)
@@ -30,6 +38,14 @@ v0.14.21
3038
face-centered (and viceversa). These functions only work for vertical and
3139
extruded spaces.
3240

41+
- We've added new convenience constructors for spaces PR [2082](https://github.com/CliMA/ClimaCore.jl/pull/2082). Here are links to the new constructors:
42+
- [ExtrudedCubedSphereSpace](https://clima.github.io/ClimaCore.jl/dev/api/#ClimaCore.CommonSpaces.ExtrudedCubedSphereSpace)
43+
- [CubedSphereSpace](https://clima.github.io/ClimaCore.jl/dev/api/#ClimaCore.CommonSpaces.CubedSphereSpace)
44+
- [ColumnSpace](https://clima.github.io/ClimaCore.jl/dev/api/#ClimaCore.CommonSpaces.ColumnSpace)
45+
- [Box3DSpace](https://clima.github.io/ClimaCore.jl/dev/api/#ClimaCore.CommonSpaces.Box3DSpace)
46+
- [SliceXZSpace](https://clima.github.io/ClimaCore.jl/dev/api/#ClimaCore.CommonSpaces.SliceXZSpace)
47+
- [RectangleXYSpace](https://clima.github.io/ClimaCore.jl/dev/api/#ClimaCore.CommonSpaces.RectangleXYSpace)
48+
3349
v0.14.20
3450
--------
3551

docs/make.jl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,7 @@ withenv("GKSwstype" => "nul") do
7777
"Installation and How-to Guides" => "installation_instructions.md",
7878
"Geometry" => "geometry.md",
7979
"Operators" => "operators.md",
80+
"Remapping" => "remapping.md",
8081
"MatrixFields" => "matrix_fields.md",
8182
"API" => "api.md",
8283
"Developer docs" => ["Performance tips" => "performance_tips.md"],

docs/src/remapping.md

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
# Remapping to regular grids
2+
3+
`ClimaCore` horizontal domains are spectral elements. Points are not distributed
4+
uniformly within an element, and elements are also not necessarily organized in
5+
a simple way. For these reasons, remapping to regular grids becomes a
6+
fundamental operations when inspecting the simulation output. In this section,
7+
we describe the remappers currently available in `ClimaCore`.
8+
9+
Broadly speaking, we can classify remappers in two categories: conservative, and
10+
non-conservative. Conservative remappers preserve areas (and masses) when
11+
interpolating from the spectral grid to Cartesian ones. Conservative remappers
12+
are non-local operations (meaning that they require communication between
13+
different elements) and are more expensive, so they are typically reserved to
14+
operations where physical conservation is important (e.g., exchange between
15+
component models in a coupled simulation). On the other hand, non-conservative
16+
remappers are local to an element and faster to evaluate, which makes them
17+
suitable to operations like diagnostics and plotting, where having perfect
18+
physical conservation is not as important.
19+
20+
## Non-conservative remapping
21+
22+
Non-conservative remappers are fast and do not require communication, but they
23+
are not as accurate as conservative remappers, especially with large elements
24+
with sharp gradients. These remappers are better suited for diagnostics and
25+
plots.
26+
27+
The main non-conservative remapper currently implemented utilizes a Lagrange
28+
interpolation with the barycentric formula in [Berrut2004], equation (3.2), for
29+
the horizontal interpolation. Vertical interpolation is linear except in the
30+
boundary elements where it is 0th order.
31+
32+
### Quick start
33+
34+
Assuming you have a `ClimaCore` `Field` with name `field`, the simplest way to
35+
interpolate onto a uniform grid is with
36+
```julia
37+
julia> import ClimaCore.Remapping
38+
julia> Remapping.interpolate(field)
39+
```
40+
41+
This will return an `Array` (or a `CuArray`) with the `field` interpolated on
42+
some uniform grid that is automatically determined based on the `Space` of the
43+
given `field`. To obtain such coordinates, you can call the
44+
`Remapping.default_target_hcoords` and `Remapping.default_target_zcoords`
45+
functions. These functions return an `Array` with the coordinates over which
46+
interpolation will occur. These arrays are of type `Geometry.Point`s.
47+
48+
`ClimaCore.Remapping.interpolate` allocates new output arrays. As such, it is
49+
not suitable for performance-critical applications.
50+
`ClimaCore.Remapping.interpolate!` performs interpolation in-place. When using
51+
the in-place version`, the `dest`ination has to have the same array type as the
52+
device in use (e.g., `CuArray` for CUDA runs) and has to be `nothing` for
53+
non-root processes. For performance-critical applications, it is preferable to a
54+
`ClimaCore.Remapping.Remapper` and use it directly (see next Section).
55+
56+
#### Example
57+
58+
Given `field`, a `Field` defined on a cubed sphere.
59+
60+
By default, a target uniform grid is chosen (with resolution `hresolution` and
61+
`vresolution`), so remapping is
62+
```julia
63+
interpolated_array = interpolate(field, hcoords, zcoords)
64+
```
65+
Coordinates can be specified:
66+
67+
```julia
68+
longpts = range(-180.0, 180.0, 21)
69+
latpts = range(-80.0, 80.0, 21)
70+
zpts = range(0.0, 1000.0, 21)
71+
72+
hcoords = [Geometry.LatLongPoint(lat, long) for long in longpts, lat in latpts]
73+
zcoords = [Geometry.ZPoint(z) for z in zpts]
74+
75+
interpolated_array = interpolate(field, hcoords, zcoords)
76+
```
77+
The output is defined on the Cartesian product of `hcoords` with `zcoords`.
78+
79+
If the default target coordinates are being used, it is possible to broadcast
80+
`ClimaCore.Geometry.components` to extract them as a vector of tuples (and then
81+
broadcast `getindex` to extract the respective coordinates as vectors).
82+
83+
### The `Remapper` object
84+
85+
A `Remapping.Remapper` is an object that is tied to a specified `Space` and can
86+
interpolate scalar `Field`s defined on that space onto a predefined target grid.
87+
The grid does not have to be regular, but it has to be defined as a Cartesian
88+
product between some horizontal and vertical coordinates (meaning, for each
89+
horizontal point, there is a fixed column of vertical coordinates).
90+
91+
Let us create our first remapper, assuming we have `space` defined on the surface of the sphere
92+
```julia
93+
import ClimaCore.Geometry: LatLongPoint, ZPoint
94+
import ClimaCore.Remapping: Remapper
95+
96+
hcoords = [Geometry.LatLongPoint(lat, long) for long in -180.:180., lat in -90.:90.]
97+
remapper = Remapper(space, target_hcoords)
98+
```
99+
This `remapper` object knows can interpolate `Field`s defined on `space` with the same `interpolate` and `interpolate!` functions.
100+
```julia
101+
import ClimaCore.Fields: coordinate_field
102+
import ClimaCore.Remapping: interpolate, interpolate!
103+
104+
example_field = coordinate_field(space)
105+
interpolated_array = interpolate(remapper, example_field)
106+
107+
# Interpolate in place
108+
interpolate!(interpolated_array, remapper, example_field)
109+
```
110+
111+
Multiple fields defined on the same space can be interpolate at the same time
112+
```julia
113+
example_field2 = cosd.(example_field)
114+
interpolated_arrays = interpolate(remapper, [example_field, example_field2])
115+
```
116+
117+
When interpolating multiple fields, greater performance can be achieved by
118+
creating the `Remapper` with a larger internal buffer to store intermediate
119+
values for interpolation. Effectively, this controls how many fields can be
120+
remapped simultaneously in `interpolate`. When more fields than `buffer_length`
121+
are passed, the remapper will batch the work in sizes of `buffer_length`. The optimal number of fields passed is the `buffer_length` of the `remapper`. If
122+
more fields are passed, the `remapper` will batch work with size up to its
123+
`buffer_length`.
124+
125+
#### Example
126+
127+
Given `field1`,`field2`, two `Field` defined on a cubed sphere.
128+
129+
```julia
130+
longpts = range(-180.0, 180.0, 21)
131+
latpts = range(-80.0, 80.0, 21)
132+
zpts = range(0.0, 1000.0, 21)
133+
134+
hcoords = [Geometry.LatLongPoint(lat, long) for long in longpts, lat in latpts]
135+
zcoords = [Geometry.ZPoint(z) for z in zpts]
136+
137+
space = axes(field1)
138+
139+
remapper = Remapper(space, hcoords, zcoords)
140+
141+
int1 = interpolate(remapper, field1)
142+
int2 = interpolate(remapper, field2)
143+
144+
# Or
145+
int12 = interpolate(remapper, [field1, field2])
146+
# With int1 = int12[1, :, :, :]
147+
```
148+
149+
## Conservative remapping with `TempestRemap`
150+
151+
This section hasn't been written yet. You can help by writing it.
152+
153+
TODO: finish writing this section.

ext/cuda/remapping_distributed.jl

Lines changed: 43 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,14 @@ function _set_interpolated_values_device!(
1717
)
1818
# FIXME: Avoid allocation of tuple
1919
field_values = tuple(map(f -> Fields.field_values(f), fields)...)
20-
nblocks, _ = size(interpolation_matrix[1])
21-
nthreads = length(vert_interpolation_weights)
20+
21+
purely_vertical_space = isnothing(interpolation_matrix)
22+
num_horizontal_points =
23+
purely_vertical_space ? 1 : prod(size(local_horiz_indices))
24+
num_points = num_horizontal_points * length(vert_interpolation_weights)
25+
max_threads = 256
26+
nthreads = min(num_points, max_threads)
27+
nblocks = cld(num_points, nthreads)
2228
args = (
2329
out,
2430
interpolation_matrix,
@@ -131,6 +137,41 @@ function set_interpolated_values_kernel!(
131137
return nothing
132138
end
133139

140+
# GPU, vertical case
141+
function set_interpolated_values_kernel!(
142+
out::AbstractArray,
143+
::Nothing,
144+
::Nothing,
145+
vert_interpolation_weights,
146+
vert_bounding_indices,
147+
field_values,
148+
)
149+
# TODO: Check the memory access pattern. This was not optimized and likely inefficient!
150+
num_fields = length(field_values)
151+
num_vert = length(vert_bounding_indices)
152+
153+
vindex = (blockIdx().y - Int32(1)) * blockDim().y + threadIdx().y
154+
findex = (blockIdx().z - Int32(1)) * blockDim().z + threadIdx().z
155+
156+
totalThreadsY = gridDim().y * blockDim().y
157+
totalThreadsZ = gridDim().z * blockDim().z
158+
159+
CI = CartesianIndex
160+
for j in vindex:totalThreadsY:num_vert
161+
v_lo, v_hi = vert_bounding_indices[j]
162+
A, B = vert_interpolation_weights[j]
163+
for k in findex:totalThreadsZ:num_fields
164+
if j num_vert && k num_fields
165+
out[j, k] = (
166+
A * field_values[k][CI(1, 1, 1, v_lo, 1)] +
167+
B * field_values[k][CI(1, 1, 1, v_hi, 1)]
168+
)
169+
end
170+
end
171+
end
172+
return nothing
173+
end
174+
134175
function _set_interpolated_values_device!(
135176
out::AbstractArray,
136177
fields::AbstractArray{<:Fields.Field},

src/Remapping/Remapping.jl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ using LinearAlgebra, StaticArrays
55
import ClimaComms
66
import ..DataLayouts,
77
..Geometry,
8+
..Domains,
89
..Spaces,
910
..Grids,
1011
..Topologies,

0 commit comments

Comments
 (0)