MatrixCustomizationUnitTesting

Notes on unit testing system for parallel customization system developed (begun on SEA-STL flight 6/15/08).

Directory structure:

unit-tests/

• run-unit-tests.py (sh?) add-unit-test.py (sh?) update-gold-standard.py (sh?) unit-test-Index choices/ txt-suites/ skeletons/ [tsdb skeletons]

  • Index.lisp

  home/ [tsdb home]

  • gold/ current/ [local, not under cvs]

  grammars/ [local, not under cvs] logs/ [local, not under cvs]

Notes

• Everything in the subdirectories is placed there by the scripts and named by the scripts. Language name in choices file used as the basis of the naming.

  We need a convention for them 🙂

User process

I: Using unit tests

1. Before committing changes to cvs (or at any other time),invoke run-unit-tests
2. ls -l on new directory in logs/ to see which unit tests reporteddifferences
3. Run tsdb++ to compare by hand the gold and current profiles

for those unit tests and diagnose as progress or regression.

1. Fix regressions
2. Invoke update-gold-standard to reify cases of progress (and put the new gold standard profiles under cvs control).
3. Be sure to run cvs commit such that any new gold standards are committed. (e.g., in the unit-tests/ directory, or above.)

II: Creating additional unit tests

1. Develop choices files and corresponding txtsuites for unit tests, and run by hand in tsdb++ to make sure they do what you want, and also to get an initial gold standard profile.
2. Invoke make-unit-test to add to system.
3. Invoke create-initial-gold-standard to put initial gold standard profile into cvs control.
4. cvs commit.

III: Maintaining unit tests

Changes to choices file formats should be reflected.

Pseudocode and specifications

unit-test-Index is probably a flat file with one line per unit test, including: language name and comment. The choices file name, txtsuite name and skeleton name should be derivable from the language name.

run-unit-tests.py (sh?)

• for each language in unit-test-Index:

  • check for gold standard profile, choices file, and skeleton (else fail) create directory under logs/ for logs (name is time-date stamp) invoke customize.py with choices file to create grammar and

    • store in grammars/

    invoke tsdb and:

    • load grammar create instance from skeleton in home/current/ with

      • pathname based on language name

      process all items compare to gold standard for same choices file

      • check number of analyses/item check for new/lost MRSs on any item if there are any changes:

        • write details to log file write unit test comment to log file

        else:

        • create empty log file

add-unit-test.py (sh?)

• inputs: choices file, txtsuite file get language name from choices file check that it isn't already in use in unit tests check that it conforms to convention? prompt user for comment on purpose of unit test (e.g., "fluid-S case system") store choices file in choices/ with appropriate file name store txtsuite file in txtsuites/ with appropriate file name modify unit-test-Index to add languge name and comment. invoke tsdb

  • create skeleton

  modify skeletons/Index.lisp

  • (use same comment here)

  remind user to run create-initial-gold-standard

update-gold-standard.py (sh?)

• inputs: language name (to identify choices file) validate language name

  • check for choices, skeleton, gold standard check for profile in home/current check for log file showing change (i.e., non-empty) ?check that profile in home/current plausibly from same run

    • as log file (time/date)

    ?check that profile in home/current isn't too old

  double-check with user by displaying contents of log file rm -rf on previous gold standard cp -r home/current/lg home/gold/

create-initial-gold-standard.py (sh?)

• inputs: language name (to identify choices file), tsdb profile if possible, verify language name by looking inside tsdb profile (maybe in run?) check that no gold profile by that name already exists, else fail cp -r profile /home/gold/language-name