Retry regressions in workers automatically

This is easier to implement in workers rather than in the agent because
we don't need to keep track of the run count or concatenate logs within
the database. It does mean we get slightly weaker fault isolation
between runs (e.g. if one machine is having a bad day or low on memory
or whatever), but this is still an improvement.

This also means that the time taken per-regression increases, though an
exact factor is hard to work out (we never retry baseline builds, and we
only retry the first failed step...). Somewhere less than 5x though. In
practice the expectation is that most of our runs have very few
regressions (order of 1000 failed steps) compared to ~500,000 total
steps executed. That is ~0.2%. So even a large increase here is likely
to be acceptable.

If there's a large amount of regressions a single crater run may run
really slowly -- but probably it is also not very useful as a run, so we
may want some early-exit clause or confirmation step in that case
anyway. For now not too worried about that case.

Author: Mark-Simulacrum

src/runner/worker.rs

@@ -173,7 +173,40 @@ impl<'a, DB: WriteResults + Sync> Worker<'a, DB> {
             let mut result = Ok(());
             for task in tasks {
                 if result.is_ok() {
-                    result = self.run_task(&task);
+                    let max_attempts = 5;
+                    for run in 1..=max_attempts {
+                        result = self.run_task(&task);
+
+                        // We retry task failing on the second toolchain (i.e., regressions). In
+                        // the future we might expand this list further but for now this helps
+                        // prevent spurious test failures and such.
+                        //
+                        // For now we make no distinction between build failures and test failures
+                        // here, but that may change if this proves too slow.
+                        let mut should_retry = false;
+                        if result.is_err() && self.ex.toolchains.len() == 2 {
+                            let toolchain = match &task.step {
+                                TaskStep::Prepare | TaskStep::Cleanup => None,
+                                TaskStep::Skip { tc }
+                                | TaskStep::BuildAndTest { tc, .. }
+                                | TaskStep::BuildOnly { tc, .. }
+                                | TaskStep::CheckOnly { tc, .. }
+                                | TaskStep::Clippy { tc, .. }
+                                | TaskStep::Rustdoc { tc, .. }
+                                | TaskStep::UnstableFeatures { tc } => Some(tc),
+                            };
+                            if let Some(toolchain) = toolchain {
+                                if toolchain == self.ex.toolchains.last().unwrap() {
+                                    should_retry = true;
+                                }
+                            }
+                        }
+                        if !should_retry {
+                            break;
+                        }
+
+                        log::info!("Retrying task {:?} [{run}/{max_attempts}]", task);
+                    }
                 }
                 if let Err((err, test_result)) = &result {
                     if let Err(e) = task.mark_as_failed(