Testing Generated Protocol Code

The CMake project generated by the commsdsl2comms utility has a test folder containing test application(s).

At this moment there is only one available application called <proj_name>_input_test. It reads the binary data from standard input and prints to standard output detected messages and values of their fields in human-readable form. The application is suitable for fuzz testing with AFL.

In order to enable the build of the test application(s) there is a need to enable OPT_BUILD_TEST option in CMake configuration.

Linux Environment

$> cd /generated/project/dir
$> mkdir build.test && cd build.test
$> cmake -DCMAKE_INSTALL_PREFIX=${PWD}/install -DOPT_BUILD_TEST=ON ..
$> make install

Windows Environment

$> cd C:\generated\project\dir
$> mkdir build.test && cd build.test
$> cmake -G "NMake Makefiles" \
     -DCMAKE_INSTALL_PREFIX=%cd%/install -DOPT_BUILD_TEST=ON ..
$> nmake install

It is recommended to use non-release build configuration to have all the assert() checks being active.

The CMake configuration also receives various variables which allow extra configuration of the test applications. For example, the CMakeLists.txt of the cc.demo1.generated project lists the following variables:

# OPT_TEST_OPTIONS - Class name of the options for test applications, defaults to demo1::options::DefaultOptions.
# OPT_TEST_INTERFACE - Class name of the interface for test applications, defaults to demo1::Message.
# OPT_TEST_FRAME - Class name of the frame for test applications, defaults to demo1::frame::Frame.
# OPT_TEST_INPUT_MESSAGES - All input messages bundle for test applications, defaults to demo1::input::AllMessages.

It is possible to use these variable to change the default configuration of the test application. For example it is possible to compile "bare-metal" configuration of the protocol that does NOT use dynamic memory allocation and substitutes some problematic storage types such as std::string or std::vector to the equivalents comms::util::StaticString or comms::util::StaticVector provided by the COMMS library itself.

Linux Environment

$> cd /generated/project/dir
$> mkdir build.test && cd build.test
$> cmake -DCMAKE_INSTALL_PREFIX=${PWD}/install -DOPT_BUILD_TEST=ON \
     -DOPT_TEST_OPTIONS=test1::options::BareMetalDefaultOptions ..
$> make install

Windows Environment

$> cd C:\generated\project\dir
$> mkdir build.test && cd build.test
$> cmake -G "NMake Makefiles" -DCMAKE_INSTALL_PREFIX=%cd%/install 
    -DOPT_TEST_OPTIONS=test1::options::BareMetalDefaultOptions -DOPT_BUILD_TEST=ON ..
$> nmake install

The FUZZ testing can be easily performed with AFL

$> CC=afl-gcc CXX=afl-g++ cmake -DCMAKE_INSTALL_PREFIX=${PWD}/install \
    -DOPT_BUILD_TEST=ON ..

The AFL requires having initial input to work with. The simplest way is to create a single binary file with random input (even single 0 byte will do).

$> printf "\x00" > ./input/zero.bin

It may take several minutes to AFL but eventually it will be able to find binary data that leads to actual messages and exercising real fields' serialization.