Multi-aspect flexible tests

[jfgigand]

Hi,

geocoder-tester takes a "Manichaean" approach, where each test address will either pass or fail, while the possibility to select specific test files and directories provides a good degree of test perimeter.

The question here is about test depth:

• is it possible to test the geo coordinates only and not the address components?
• is it possible to have custom expected properties which may be checked or not, depending on some runtime parameter?

More generally, what do you think about having different levels of testing that can be specified at runtime?

[yohanboniface]

Why not.
An optional --fields=name,xxx may allow to run with only checking those fields.
The tricky part may be to tweak the test collector to only consider the good tests (and then of course only checking those fields when actually running them).

[jfgigand]

Yes, this looks like a good idea.
I should think more about the kind of testing we need in the long run, before choosing the approach.
Will keep you posted...