ICST-2022-submission: JavaScript Instrumentation for Search-Based Software Testing: A Study with RESTful APIs

In this package, we provide necessary information for replicating the experiment in the paper that includes

• EvoMaster: tools described in the paper with our js instrumentation and js test writer.
• jar: runnable jar for EvoMaster.
• EMB-js: five NodeJS case studies employed for the experiment.
• build.py: a python script to build tools and all five case studies.
• run.sh and runBB.sh: examples to run our tool on ncs-js case study.
• Results: raw data.
• r-script: a R script to analyze results and generate table and figures reported in the paper.

Implementation clarification

To enable SBST for NodeJs application, we implemented

• javascript instrumentation as Babel plugin, and the source code is locatedEvomaster\client-js\evomaster-client-jshere.
• driver to communicate with EvoMaster and SUT.
• JavaScript test writer for producing jest tests.

Quick build and run

Step 1. In this repo, we provide a python script to build tools and all five case studies.

Go to the root, run

python3 build.py

After the build is finished, for each of case studies, there would be a new build folder which contains instrumented code. For EvoMaster, its jar file would be found under EvoMaster/core/target.

Step 2. To try our tool with ncs-js case study, run bash

./run.sh

The script includes two settings i.e., JS-MIO and JS-Random as described in the paper. After the execution is done, you will see a folder named example which contains

• EM_JS_MIO_Test.js : a test generated by JS-MIO with fitness evaluation on 100,000 HTTP calls
• EM_JS_Random_Test.js: a test generated by JS-Random with fitness evaluation on 100,000 HTTP calls
• statistics_ncs_js.csv: statistics info (like covered targets, covered lines, covered branches) for two settings
• snapshot_ncs_js.csv: snapshot of statistics collected (at e.g., every 5% budget used) during the search for two settings

Step 3. In addition, we also provide an example for presenting coverage collection with c8 for a whole process. The example is with BB_ft setting on ncs-js case study. NOTE Before running the example, please install c8, see https://github.com/bcoe/c8. Then start the example with

./runBB.sh

After the execution is done, you will see

In example folder

• EM_BB_ft_Test.js is generated for a test generated by BB_ft setting with 3.97minutes time budget
• statistics_ncs_js.csv: there are additional info for BB_ft setting
• snapshot_ncs_js.csv: there are additional info for BB_ft setting

In EMB-js/rest/ncs, there would be a new folder named coverage which contains

• coverage-summary.json: a coverage report with json format.

Results and Analysis

Raw data in our experiment are provided in results folder

• compressedData.zip and snapshotCompressedData.zip are compressed statistics and snapshot files for white-box experiment;
• wb-exp contains coverage reports based on generated tests;
• bb-exp contains coverage reports which are collected during a whole process as the runBB.sh example.

We also provide a R script js-exp.R to analyze the result and produce tables and figures used in the paper. After run js-exp.R, in results/generatedfiles,

• table_wbtargets_all.tex is Table II for RQ1;
• for RQ1, we also provide plot-lines based on the number of covered targets over the course of the search for each of cases studies, i.e., plot_wbtargets_cyclotron-js.pdf (cyclotron-js),plot_wbtargets_disease-sh-api-js.pdf(disease-sh-api), plot_wbtargets_ncs-js.pdf (ncs-js), plot_wbtargets_nestjs-realworld-example-app-js.pdf (realworld-app-js), and plot_wbtargets_scs-js.pdf(scs-js);
• table_avg_coverage.tex is Table III for RQ2 and RQ3;
• table_avg_faults.tex is Table IV for RQ2 and RQ3;
• table_pair.tex is Table V for RQ3.

Build and run step by step

Build JavaScript Instrumentation

Go to Evomaster\client-js\evomaster-client-js, run

• step 1npm install
• step 2 npm run build

Build EvoMaster

Go to Evomaster, run

• step 3 mvn clean install -DskipTests

Instrument SUT

Go to a sut, e.g., EMB-js\rest\ncs-js, run

• step 4 npm install
• step 5 npm run build

then under the sut, you will see a new folder named build which contains instrumented code, e.g., EMB-js\rest\ncs-js\build

Start EM Driver

Go to the sut, e.g., EMB-js\rest\ncs-js

• step 6 npm run em

Start EvoMaster

• step 7 java -jar evomaster.jar

In this experiment, we use configurations like,

• --algorithm=MIO for algorithm setting;
• --outputFormat=JS_JEST for output format setting;
• --stoppingCriterion=FITNESS_EVALUATIONS --maxActionEvaluations=100000 for fitness budget setting;
• --maxTime for time budget setting;
• --blackbox for white-box and black-box setting.
• --bbSwaggerUrl for accessing schema of REST in black-box setting.