|
| 1 | +# Difftests |
| 2 | + |
| 3 | +Difftests verify correctness by running multiple implementations of the same logic and |
| 4 | +comparing their outputs. Instead of relying on fixed reference outputs, they detect |
| 5 | +discrepancies across implementations. |
| 6 | + |
| 7 | +## How It Works |
| 8 | + |
| 9 | +1. **Test Discovery** |
| 10 | + |
| 11 | + - The harness scans `tests/` for test cases. |
| 12 | + - A test case is a directory containing at least two Rust binary packages / variants |
| 13 | + to compare. |
| 14 | + |
| 15 | +2. **Configuration & Execution** |
| 16 | + |
| 17 | + - The harness creates a temporary output file for each variant binary. |
| 18 | + - The harness runs each test variant binary with `cargo run --release`. |
| 19 | + - A JSON config file with the output path and other settings is passed to the test |
| 20 | + binary as `argv[1]`. |
| 21 | + - The binary reads the config, runs its computation, and writes to the output file. |
| 22 | + - Tests are run serially so they have full control of the GPU. |
| 23 | + |
| 24 | +3. **Output Comparison** |
| 25 | + - The harness reads outputs as opaque bytes. |
| 26 | + - If outputs differ, the test fails. |
| 27 | + |
| 28 | +Because the difftest harness merely runs Rust binaries in a directory, it supports |
| 29 | +testing various setups. For example, you can: |
| 30 | + |
| 31 | +- Compare different CPU host code (`ash`, `wgpu`, etc.) with different GPU backends |
| 32 | + (`rust-gpu`, `cuda`, `metal`, `wgsl`, etc.) against each other to make sure the output |
| 33 | + is consistent. |
| 34 | +- Verify that CPU and GPU implementations produce the same output. |
| 35 | +- Ensure the same `rust-gpu` code gives identical results across different dispatch |
| 36 | + methods. |
| 37 | + |
| 38 | +## Writing a Test |
| 39 | + |
| 40 | +Create a subdirectory under `tests/` with the test name. For example, `tests/foo/` for a |
| 41 | +test named `foo`. In the test directory, create 2 or more Rust binary packages. Add the |
| 42 | +packages to the top-level workspace `Cargo.toml` in the `tests/` directory. _Note that |
| 43 | +this isn't the top-level workspace for the project._ The test binaries are in their own |
| 44 | +workspace rather than the main workspace. |
| 45 | + |
| 46 | +### Test Binary Example |
| 47 | + |
| 48 | +Each test binary must: |
| 49 | + |
| 50 | +1. Have a unique package name in `Cargo.toml`. |
| 51 | +2. Read the config file path from `argv[1]`. |
| 52 | +3. Load the config using `difftest::Config::from_path`. |
| 53 | +4. Write its computed output to `output_path`. |
| 54 | + |
| 55 | +For example: |
| 56 | + |
| 57 | +```rust |
| 58 | +use difftest::config::Config; |
| 59 | +use std::{env, fs, io::Write}; |
| 60 | + |
| 61 | +fn main() { |
| 62 | + let config_path = env::args().nth(1).expect("No config path provided"); |
| 63 | + let config = Config::from_path(&config_path).expect("Invalid config"); |
| 64 | + |
| 65 | + // Real work would go here. |
| 66 | + let output = compute_test_output(); |
| 67 | + |
| 68 | + let mut file = fs::File::create(&config.output_path) |
| 69 | + .expect("Failed to create output file"); |
| 70 | + file.write_all(&output).expect("Failed to write output"); |
| 71 | +} |
| 72 | +``` |
| 73 | + |
| 74 | +### Common test types |
| 75 | + |
| 76 | +Of course, many test will have common host and GPU needs. Rather than require every test |
| 77 | +binary to reimplement functionality, we have created some common tests with reasonable |
| 78 | +defaults in the `difftest` library. |
| 79 | + |
| 80 | +For example, this will handle compiling the current crate as a Rust compute shader, |
| 81 | +running it via `wgpu`, and writing the output to the appropriate place: |
| 82 | + |
| 83 | +```rust |
| 84 | +fn main() { |
| 85 | + // Load the config from the harness. |
| 86 | + let config = Config::from_path(std::env::args().nth(1).unwrap()).unwrap(); |
| 87 | + |
| 88 | + // Define test parameters, loading the rust shader from the current crate. |
| 89 | + let test = WgpuComputeTest::new(RustComputeShader::default(), [1, 1, 1], 1024); |
| 90 | + |
| 91 | + // Run the test and write the output to a file. |
| 92 | + test.run_test(&config).unwrap(); |
| 93 | +} |
| 94 | +``` |
| 95 | + |
| 96 | +and this will handle loading a shader named `shader.wgsl` or `compute.wgsl` in the root |
| 97 | +of the current crate, running it via `wgpu`, and writing the output to the appropriate |
| 98 | +place: |
| 99 | + |
| 100 | +```rust |
| 101 | +fn main() { |
| 102 | + // Load the config from the harness. |
| 103 | + let config = Config::from_path(std::env::args().nth(1).unwrap()).unwrap(); |
| 104 | + |
| 105 | + // Define test parameters, loading the wgsl shader from the crate directory. |
| 106 | + let test = WgpuComputeTest::new(WgslComputeShader::default(), [1, 1, 1], 1024); |
| 107 | + |
| 108 | + // Run the test and write the output to a file. |
| 109 | + test.run_test(&config).unwrap(); |
| 110 | +} |
| 111 | +``` |
| 112 | + |
| 113 | +## Running Tests |
| 114 | + |
| 115 | +### Run all difftests: |
| 116 | + |
| 117 | +```sh |
| 118 | +cargo difftest |
| 119 | +``` |
| 120 | + |
| 121 | +Note that `cargo difftest` is an alias in `.cargo/config` for `cargo run --release -p |
| 122 | +difftest --`. |
| 123 | + |
| 124 | +### Run specific tests by name: |
| 125 | + |
| 126 | +```sh |
| 127 | +cargo difftest some_test_name |
| 128 | +``` |
| 129 | + |
| 130 | +### Show stdout/stderr from tests: |
| 131 | + |
| 132 | +```sh |
| 133 | +cargo difftest --nocapture |
| 134 | +``` |
| 135 | + |
| 136 | +## Debugging Failing Tests |
| 137 | + |
| 138 | +If outputs differ, the error message lists: |
| 139 | + |
| 140 | +- Binary package names |
| 141 | +- Their directories |
| 142 | +- Output file paths |
| 143 | + |
| 144 | +Inspect the output files with your preferred tools to determine the differences. |
| 145 | + |
| 146 | +If you suspect a bug in the test harness, you can view detailed test harness logs: |
| 147 | + |
| 148 | +```sh |
| 149 | +RUST_LOG=trace cargo difftest |
| 150 | +``` |
0 commit comments