|
| 1 | +# Tests |
| 2 | + |
| 3 | +Our test suite organization and goals are outlined here. This document was designed to be somewhat agnostic to this package. |
| 4 | + |
| 5 | +Some goals we hope to achieve are: |
| 6 | + |
| 7 | + - Achieve high coverage in our unit tests |
| 8 | + - Run our unit tests in a very short time (<3 minutes) |
| 9 | + |
| 10 | +In order to achieve thes goals, we need to strategically organize our tests, and partition unit tests from more complicated ones. To do so, we've organized our tests by "levels": |
| 11 | + |
| 12 | +## Test levels |
| 13 | + |
| 14 | + - Level 1: Quick and high coverage unit tests (with only `Float64`). These tests are run with `MyPackage> test`, and should be as quick as possible, while exercising, and testing the logic of, a high percentage of the source code. |
| 15 | + |
| 16 | + - Level 2: Inference, flop-counting, and allocation tests. Since compilation incurs allocations, allocation _tests_ require running everything twice, in order to measure runtime allocations. Inference tests can be somewhat expensive, too, as this involves verifying inference. This includes, for example, running tests with `Float32`. Therefore, these tests are separate from Level 1. |
| 17 | + |
| 18 | + - Level 3: Convergence tests. Convergence tests are highly valuable in testing the correctness of algorithms with theoretic convergence guarantees. This requires running code multiple times (often more than 2 or 3), therefore, these tests are separate from Level 1 and 2. |
| 19 | + |
| 20 | + - Level 4: Examples and simulations. Computing solutions to ODEs/PDEs can be very expensive, and are separated into level 4 tests. These may be examples that one can compare with (approximate or) exact solutions, or with (e.g., numerical) data. |
| 21 | + |
| 22 | + - Benchmarks: Benchmarks are special in that they must be performed at target resolutions, and we often can't strictly test benchmarks due to variation / noise (which can easily reach 30%). Therefore, benchmarks will only run and log the results, so that we have a trail of their results over time. |
| 23 | + |
| 24 | +## Test organization |
| 25 | + |
| 26 | +We have separate modules for different components, and now different test levels. Partitioning our tests requires some level of organization. |
| 27 | + |
| 28 | + - Tests are firstly organized in folders based on the package's modules. e.g., `MyPackage.SubModuleA` -> `test/SubModuleA/`. |
| 29 | + - Files use the following prefix convention: |
| 30 | + - `utils_` - non-test files. These files should only define methods, and should be designed for flexibility (e.g., can create objects with `Float32` or `Float64`). |
| 31 | + - `unit_` - Unit test files |
| 32 | + - `inference_` - Inference test files |
| 33 | + - `allocs_` - Allocation test files |
| 34 | + - `convergence_` - Convergence test files |
| 35 | + - `benchmark_` - Benchmark test files |
| 36 | + |
| 37 | +This should help keep including shared files between `unit`, `inference` etc. tests simple as they are still organized by the packages sub-modules. |
| 38 | + |
| 39 | +One helpful tip on organizing tests is: try to (sensibly) split off the creation of the method arguments from testing a particular method. This is very helpful for REPL testing, and will allow developers to share lots of code between the different levels of tests. |
| 40 | + |
| 41 | +# ClimaCore specifics: |
| 42 | + |
| 43 | + - Our benchmark tests must be performed at a target resolution. |
| 44 | + |
0 commit comments