regressiontests

Basics

• A regression test consists of a test workspace, which writes arbitrary output data to disk (usually images and volumes), and a set of reference data files. On each test run the generated output data of the workspace is compared to the reference data.
• Regression tests are associated with modules: Test workspaces are located in the test/ subdirectory of the respective module's base directory (e.g. voreen/modules/base/test/)
  • Rendering tests, i.e., test workspaces whose processors access the OpenGL API, must be stored in test/render/!
  • All tests outside the render/ subdirectory are executed without OpenGL context!
• Input and reference data is stored in a separate repository:
  • The repository contains a single directory voreen-testdata/, which is referred to as test data base directory in the following
  • voreen-testdata/input contains the input data a test workspace can use
    • Test workspaces are executed within the test data base directory, i.e., all paths to input files must be relative to this directory
  • voreen-testdata/reference contains the tests' reference data
    • Each module has its own reference data subdirectory voreen-testdata/reference/<modulename>/
    • Each test has its own reference data subdirectory, whose path is derived from the test file's name and its location. Examples:
      • Test file: voreen/modules/mymodule/test/mytest.vws Ref. dir: voreen-testdata/reference/mymodule/mytest/
      • Test file: voreen/modules/base/test/render/background.vws Ref. dir: voreen-testdata/reference/base/render/background/
  • The test data repository is now a submodule of the main repository. Instead of cloning the voreen-testdata repository manually, you may:
    • Run git submodule update --init --recursive to initialize it.
    • Run git submodule update --remote to update the revision, the submodule points to in the main repository. This may be required after pushing changes to the voreen-testdata repository manually.
• A test workspace is expected to write its output data to the subdirectory output/

Step-by-step Guide for Adding a Test

Preparations

• Clone test data repository (see above)
• Set test data base directory in VoreenVE application settings

Adding a Test

1. Construct test workspace

• Use input data from voreen-testdata/input/
• Configure output processors (Canvas, VolumeSave, GeometrySave, ...) to write their data to voreen-testdata/output/
  • Write out data to use as reference data

2. Save test workspace to the module's test/ subdirectory (see above)

• When asked whether to use the test data base directory as working directory for the workspace, select "yes"

3. Copy generated reference data from voreen-testdata/output/ to the test's reference directory (see above)
4. Do not forget to push reference data (and new input data, if added) to test data repository, before pushing the test case to the main directory

Optional Config File

Optionally, it is possible to create a config file for each test workspace. This has to have the exact same name as the workspace file does, but needs to end with .cfg

Example naming:

geometrysourcesave.vws
geometrysourcesave.cfg

Example config file content:

<?xml version="1.0" ?>
<VoreenData version="1">
    <TestCaseConfiguration>
        <enabled value="true" />
        <ignored value="false" />
        <voxelDiffTolerance value="0.000" />
        <maxErrorVoxels value="0" />
    </TestCaseConfiguration>
</VoreenData>

The following attributes are supported:

name	value	default	meaning
enabled	true/false	true	Enables/disables the testcase.
ignored	true/false	false	Determines, whether testcase will be ignored. If test case is enabled however, it will be executed but the result will be ignored.
timeout	positive decimal	0	Maximum execution time of the test case in seconds (unix only).
pixelDiffTolerance	decimal within range [0, 1]	0.2	Maximum allowed per-pixel color distance in image comparisons (RGBA color space).
maxErrorPixels	positive integer	50	Maximum allowed number of pixels with a difference above the tolerance.
pixelSearchNeighborhood	positive integer	1	Radius of the neighborhood around a pixel to search for a match in the reference image.
voxelDiffTolerance	decimal within range [0, 1]	1e-06	Maximum allowed per-voxel difference in volume comparisons.
maxErrorVoxels	positive integer	0	Maximum allowed number of voxels with a difference above the tolerance.
geometryDiffTolerance	positive decimal	1e-06	Maximum allowed vertex distance in geometry comparisons.

Testing Guidelines

• Use one test case per processor and name the test file after the processor it's testing.
• Tests should be as close to unit tests as possible:
  • A test network should only contain the processors that are necessary for testing the functionality of the processor under test. In the optimal case this is just: DataSource -> ProcessorUnderTest -> DataSink
  • Data processing tests should not involve OpenGL rendering.
  • Unless depth values are required, image processor tests should use an ImageSource instead of a full-blown volume rendering pipeline as data supplier.
• Use PNG instead of JPG as output image format, in order to keep transparency information.
• Minimize data size:
  • Only add new input data, if really necessary.
  • For volume processing tests, the smallest walnut data sets (walnut_uint8, walnut_unit16, walnut_float) are usually sufficient as input data. Using larger input data sets usually results in larger output files.
  • For rendering tests, output image dimensions of about 400x400 pixels are usually sufficient.

Executing Tests Locally

• Build regression test application (apps/tests/regressiontest), also requires voreentool (apps/voreentool)
• Required options:
  • --testcases <testPath0> <testPath1> ... List of test paths containing the test cases to execute. Each test path may either specify a concrete test file or a directory and has the form <module>[/<testCaseSubPath>] where <testCaseSubPath> is relative to the module's test directory. Examples:
    • --testcases all - runs all test cases
    • --testcases base - runs all test cases of the base module
    • --testcases base/render - runs all test cases in modules/base/test/render/
    • --testcases base/render/background.vws - runs the single test case modules/base/test/render/background.vws
  • --datadir <path> - path to the test data base directory
• Some further options:
  • --reportdir <path> - directory where the tests' output data and log files will be stored
  • --htmlReport <htmlFile> - generates an HTML report of the test run
  • --useCaching false - You probably want this unless you really know what you are doing (or are fine with being confused while debugging broken processors that do caching).
  • --help - guess what ;)
• Sample call:

regressiontest.exe --useCaching false --testcases all --datadir D:\voreen-testdata\voreen-testdata --reportdir D:\voreen-testdata\report --htmlReport D:\voreen-testdata\report\report.html