fuzz testing

Background

Fuzzers are computer programs that are used to detect security holes in other programs, by sending those programs inputs that may fall outside the expected inputs, and thus reveal subtle bugs in the code. Most such fuzzers are driven by code coverage. Some fuzzers learn from previous inputs and results to produce inputs that exercise the code under test more effectively. DeepState is a testing framework that allows easy testing of C and C++ programs with sophisticated fuzzers. R has some simple random testers, but no coverage-driven fuzzers that learn to produce interesting inputs.

Related work

• fuzzr provides less sophisticated random testing of R functions. (and no framework for writing unit tests)
• covr provides instrumentation of R code for unit testing. (instrumentation is required for more sophisticated fuzzers to learn a function from inputs to the execution path through the code)
• testthat provides functions for defining tests, similar to deepstate.

GSOC coding project: “DeepState” for R

The goal of this GSOC project is to implement new features for defining unit tests in R code with coverage-driven fuzzing support. If feasible, a DeepState-like approach that allows multiple back-end fuzzers would be good; however, instrumentation that would make this useful may be hard to achieve. For an example of such an effort, see python-afl, which instruments non-compiled Python code so that AFL, a well-known and very successful fuzzer, can test Python.

After implementing the framework, we will use it to detect bugs in several widely used R packages (including base R).

TODO more detailed goals.

Expected impact

Currently R package developers do not systematically use random testing on their code; this project would make it much easier to do so. If we end up detecting new bugs in R packages, it will result in improvements to these packages.

Mentors

Please get in touch with Toby Dylan Hocking <toby.hocking@r-project.org> and Alex Groce <alex.groce@nau.edu> after completing at least one of the tests below.

Tests

Do one or several — doing more hard tests makes you more likely to be selected.

• Easy: use fuzzr on one of your favorite R functions, and post the results to a gist.
• Medium: TODO
• Hard: TODO

Solutions of tests

• Students, please post a link to your test results here.

  Name: Ronnie Gandhi

  Email: gronnie@cs.iitr.ac.in

  Website: RonnieGandhi

  University: Indian Institute of Technology, Roorkee

  Course: Computer Science and Engineering

  Solution to Easy Test: Easy

  Solution to Medium Test: TODO

  Solution to Hard Test: TODO