From 645ec69b3ddfe7160b70c87aa0119eceee420c9d Mon Sep 17 00:00:00 2001 From: Eric Peterson Date: Tue, 26 Nov 2024 22:04:46 -0800 Subject: [PATCH 1/2] Added initial testing documentation --- tests/README.txt | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 tests/README.txt diff --git a/tests/README.txt b/tests/README.txt new file mode 100644 index 0000000..8104d8d --- /dev/null +++ b/tests/README.txt @@ -0,0 +1,43 @@ +This document describes the testing done in the OSIPI TF2.4 IVIM repository + + +-- Outline -- +1. Testing philosophy +2. Testing structure +3. Testing results + + + +-- Testing philosophy -- +Testing is integral to the repository. +There are many different contributions from many different people and only through diligent testing can they all be ensured to work correctly together. +There are 3 major types of tests we conduct. +1. Requirements + - Which are required to pass + - All algorithms have the same requirements + - E.g. bounds honored, code runs reasonably +2. Expectations + - Considered warnings + - Should not necessarily prevent a merge + - Should cover performance changes + - E.g. performance changes +3. Performance + - The accuracy of the results + - The speed of the generated results + +-- Testing structure -- +* The testing is controlled in several places. +* The testing itself is done with pytest which parses files for "test_" and runs the appropriate tests. +* The pytest testing can be done on your own machine by running "python -m pytest". +** This is configured with pytests.ini and conftest.py +* Testing on github is controlled by the github actions which are in the workflows folder. +* Each workflow performs a series of tests, and is defined by the yml file. +* Each workflow can run at specified times and with specified outputs. +* Currently the major testing workflows are unit_test.yml and analysis.yml. +* The unit_test workflow is done frequently and is relativly fast and does some basic algorithm testing. +* The analysis workflow is done more infrequently with merges to the main branch and does more Expecation testing. + + +-- Testing results -- +The test results are written to several files, most notably an xml file for machine parsing, and a web page for code coverage. +The "analysis" tests are written to a csv file as well. From 3d4d664b69af4a9e1b80198c8d66e40155528271 Mon Sep 17 00:00:00 2001 From: Eric Peterson Date: Wed, 27 Nov 2024 07:31:30 -0800 Subject: [PATCH 2/2] Updates from meeting --- tests/README.txt | 46 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 40 insertions(+), 6 deletions(-) diff --git a/tests/README.txt b/tests/README.txt index 8104d8d..db49243 100644 --- a/tests/README.txt +++ b/tests/README.txt @@ -11,21 +11,55 @@ This document describes the testing done in the OSIPI TF2.4 IVIM repository -- Testing philosophy -- Testing is integral to the repository. There are many different contributions from many different people and only through diligent testing can they all be ensured to work correctly together. +Automated testing happens on different platforms and versions. There are 3 major types of tests we conduct. 1. Requirements - - Which are required to pass + - Runs on each commit + - Must pass to merge - All algorithms have the same requirements - - E.g. bounds honored, code runs reasonably + - E.g. bounds honored, code runs reasonably, properly integrated + - Would prevent a merge if not passing + - Flexibile input/output + - Categories for testing + -- Contributions - Some aspects currently tested as unit_test + --- Initial bounds are respected + ---- Needs implemening + --- Initial guess is respected + ---- Needs implemening - may no be possible + --- Runs reasonably + ---- Needs implemening: reduced data size, broadened limits + --- Contains information about algorithm + -- Wrapper + --- Initial guess is in bounds + --- Reasonable default bounds - f: [0 1], D >= 0 & D < D*, D* >= 0 + --- Input size is respected - result is same size as input + --- Dictionary is returned - worth explicit testing? + -- Phantom - lower priority + --- Data can be pulled 2. Expectations + - Run on each merge - Considered warnings - Should not necessarily prevent a merge - - Should cover performance changes - - E.g. performance changes -3. Performance - - The accuracy of the results + - Categories for testing + -- Determine performance changes from reference run + --- Currently implemented but could be made easier to interact with + --- Could be made easier and faster +3. Characterization + - Run on demand + - Performance of the algorithms + - The accuracy and precision of the results - The speed of the generated results + - Human readable report of the wrapped algorithms + - Categories for testing + -- Simulations + --- Voxels from tissue and characterize algorithms + --- Visualize parameter maps + -- True data + --- Visualize parameter maps + --- Correlations between algorithms - plot the results and differences -- Testing structure -- + * The testing is controlled in several places. * The testing itself is done with pytest which parses files for "test_" and runs the appropriate tests. * The pytest testing can be done on your own machine by running "python -m pytest".