Acceptance tests with many permutations

[corneil]

I thought I'd share what we've been working on. For now I'm using a combination of HorizontalAutoScaler with a kick in the but. I scale up the runner cluster before increasing the maxReplicas.
If anything fails I leave the clusters in place so that a re-run can be done or some other investigation.
I have a Housekeeping job to scale down everything if there is no activity for a period.