Auto merge of #687 - Mark-Simulacrum:retry-regressions, r=Mark-Simulacrum

Retry regressions in workers automatically

This is easier to implement in workers rather than in the agent because we don't need to keep track of the run count or concatenate logs within the database. It does mean we get slightly weaker fault isolation between runs (e.g. if one machine is having a bad day or low on memory or whatever), but this is still an improvement.

This also means that the time taken per-regression increases, though an exact factor is hard to work out (we never retry baseline builds, and we only retry the first failed step...). Somewhere less than 5x though. In practice the expectation is that most of our runs have very few regressions (order of 1000 failed steps) compared to ~500,000 total steps executed. That is ~0.2%. So even a large increase here is likely to be acceptable.

If there's a large amount of regressions a single crater run may run really slowly -- but probably it is also not very useful as a run, so we may want some early-exit clause or confirmation step in that case anyway. For now not too worried about that case.

Author: bors

src/runner/worker.rs

@@ -173,7 +173,40 @@ impl<'a, DB: WriteResults + Sync> Worker<'a, DB> {
             let mut result = Ok(());
             for task in tasks {
                 if result.is_ok() {
-                    result = self.run_task(&task);
+                    let max_attempts = 5;
+                    for run in 1..=max_attempts {
+                        result = self.run_task(&task);
+
+                        // We retry task failing on the second toolchain (i.e., regressions). In
+                        // the future we might expand this list further but for now this helps
+                        // prevent spurious test failures and such.
+                        //
+                        // For now we make no distinction between build failures and test failures
+                        // here, but that may change if this proves too slow.
+                        let mut should_retry = false;
+                        if result.is_err() && self.ex.toolchains.len() == 2 {
+                            let toolchain = match &task.step {
+                                TaskStep::Prepare | TaskStep::Cleanup => None,
+                                TaskStep::Skip { tc }
+                                | TaskStep::BuildAndTest { tc, .. }
+                                | TaskStep::BuildOnly { tc, .. }
+                                | TaskStep::CheckOnly { tc, .. }
+                                | TaskStep::Clippy { tc, .. }
+                                | TaskStep::Rustdoc { tc, .. }
+                                | TaskStep::UnstableFeatures { tc } => Some(tc),
+                            };
+                            if let Some(toolchain) = toolchain {
+                                if toolchain == self.ex.toolchains.last().unwrap() {
+                                    should_retry = true;
+                                }
+                            }
+                        }
+                        if !should_retry {
+                            break;
+                        }
+
+                        log::info!("Retrying task {:?} [{run}/{max_attempts}]", task);
+                    }
                 }
                 if let Err((err, test_result)) = &result {
                     if let Err(e) = task.mark_as_failed(