Skip to content

Commit 2bc09fb

Browse files
Add a test README (#1784)
1 parent a7bb4ec commit 2bc09fb

File tree

1 file changed

+44
-0
lines changed

1 file changed

+44
-0
lines changed

test/README.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Tests
2+
3+
Our test suite organization and goals are outlined here. This document was designed to be somewhat agnostic to this package.
4+
5+
Some goals we hope to achieve are:
6+
7+
- Achieve high coverage in our unit tests
8+
- Run our unit tests in a very short time (<3 minutes)
9+
10+
In order to achieve thes goals, we need to strategically organize our tests, and partition unit tests from more complicated ones. To do so, we've organized our tests by "levels":
11+
12+
## Test levels
13+
14+
- Level 1: Quick and high coverage unit tests (with only `Float64`). These tests are run with `MyPackage> test`, and should be as quick as possible, while exercising, and testing the logic of, a high percentage of the source code.
15+
16+
- Level 2: Inference, flop-counting, and allocation tests. Since compilation incurs allocations, allocation _tests_ require running everything twice, in order to measure runtime allocations. Inference tests can be somewhat expensive, too, as this involves verifying inference. This includes, for example, running tests with `Float32`. Therefore, these tests are separate from Level 1.
17+
18+
- Level 3: Convergence tests. Convergence tests are highly valuable in testing the correctness of algorithms with theoretic convergence guarantees. This requires running code multiple times (often more than 2 or 3), therefore, these tests are separate from Level 1 and 2.
19+
20+
- Level 4: Examples and simulations. Computing solutions to ODEs/PDEs can be very expensive, and are separated into level 4 tests. These may be examples that one can compare with (approximate or) exact solutions, or with (e.g., numerical) data.
21+
22+
- Benchmarks: Benchmarks are special in that they must be performed at target resolutions, and we often can't strictly test benchmarks due to variation / noise (which can easily reach 30%). Therefore, benchmarks will only run and log the results, so that we have a trail of their results over time.
23+
24+
## Test organization
25+
26+
We have separate modules for different components, and now different test levels. Partitioning our tests requires some level of organization.
27+
28+
- Tests are firstly organized in folders based on the package's modules. e.g., `MyPackage.SubModuleA` -> `test/SubModuleA/`.
29+
- Files use the following prefix convention:
30+
- `utils_` - non-test files. These files should only define methods, and should be designed for flexibility (e.g., can create objects with `Float32` or `Float64`).
31+
- `unit_` - Unit test files
32+
- `inference_` - Inference test files
33+
- `allocs_` - Allocation test files
34+
- `convergence_` - Convergence test files
35+
- `benchmark_` - Benchmark test files
36+
37+
This should help keep including shared files between `unit`, `inference` etc. tests simple as they are still organized by the packages sub-modules.
38+
39+
One helpful tip on organizing tests is: try to (sensibly) split off the creation of the method arguments from testing a particular method. This is very helpful for REPL testing, and will allow developers to share lots of code between the different levels of tests.
40+
41+
# ClimaCore specifics:
42+
43+
- Our benchmark tests must be performed at a target resolution.
44+

0 commit comments

Comments
 (0)