We could use `tox` for this; another project I work on, Mimic, does it like this: https://github.com/rackerlabs/mimic/blob/master/tox.ini