Global spec/feature run ordering extension with several available run order variants

docs/extensions.adoc

@@ -44,6 +44,18 @@ runner {
 
 See the <<parallel_execution.adoc#parallel-execution,Parallel Execution>> section for a detailed description.
 
+=== Test Order Configuration
+
+[source,groovy]
+----
+runner {
+  orderer new RandomSpecOrderer()  // randomize specification run order
+}
+----
+
+Instead of the default run order, you can configure Spock to execute specifications and/or features e.g. in random,
+alphabetical or manually assigned, annotation-based order. See the <<#_run_order>> section for more details.
+
 == Built-In Extensions
 
 Most of Spock's built-in extensions are _annotation-driven_. In other words, they are triggered by annotating a
@@ -709,6 +721,114 @@ runner {
 }
 ----
 
+=== Run Order
+
+Ideally, automated tests in general and Spock specifications in particular should be independent of each other. The same
+applies to feature methods within a specification. Therefore, you should not rely on any specific order of execution
+(run order).
+
+Nevertheless, you have options to influence the run order, using the <<spock-configuration-file>> and a set of built-in
+orderers derived from super class `SpecOrderer`. Please check the Javadocs for package
+`org.spockframework.runtime.extension.builtin.orderer` for more details. You can also write your own `SpecOrderer`, if
+none of the built-in ones satisfies your needs.
+
+Please note that `@Stepwise` always trumps any run order you might have configured, i.e. `@Stepwise` "wins" against
+`SpecOrderer`.
+
+==== Random Run Order
+
+One helpful way to heuristically increase your confidence that your tests are indeed independent of each other, is to
+explicitly say goodbye to deterministic run order by randomizing it.
+
+[source,groovy]
+----
+import org.spockframework.runtime.extension.builtin.orderer.RandomSpecOrderer
+
+runner {
+  orderer new RandomSpecOrderer(
+    true,                       // Randomize overall specification run order
+    true,                       // Randomize the run order of feature methods within specifications
+    System.currentTimeMillis()  // Set a fixed value, if you want repeatable pseudo-random numbers.
+                                // This might be helpful for reproducing issues when debugging your tests.
+  )
+}
+----
+
+==== Alphabetical Run Order
+
+Less useful than random run order, but available anyway, is a way to execute specifications and/or features
+alphabetically, based on their display names and a simple `String.compareTo(String)` (no fancy locale-based collation).
+The default sorting direction is ascending, optionally you can also sort elements in descending order.
+
+[source,groovy]
+----
+import org.spockframework.runtime.extension.builtin.orderer.AlphabeticalSpecOrderer
+
+runner {
+  orderer new AlphabeticalSpecOrderer(
+    true,  // Run specifications in alphabetical order by display name
+    true,  // Run feature methods within specifications in alphabetical order by display name
+    false  // Sort in ascending order (use 'true' for descending order)
+   )
+}
+----
+
+==== Annotation-Based Run Order
+
+If you want to basically retain Spock's default run order for most or at least some of your specifications and/or
+feature methods, but modify it for particular specs/features, or take it to the extreme and manually assign run orders
+everywhere, use the `@Order(int)` annotation in combination with the annotation-based orderer:
+
+[source,groovy]
+----
+import org.spockframework.runtime.extension.builtin.orderer.AnnotatationBasedSpecOrderer
+
+runner {
+  orderer new AnnotatationBasedSpecOrderer()
+}
+----
+
+Please note, that `@Order` annotations have no effect whatsoever, if `AnnotatationBasedSpecOrderer` is not configured
+as the active orderer. E.g., you cannot expect to be able to use random ordering in combination with manually assigning
+run orders via annotations for some exceptions. Annotation-based ordering must be explicitly activated and is only
+available as a modification of Spock's default run order.
+
+Using `@Order`, the basic idea is to assume unannotated specifications and features to all carry an implicit `@Order(0)`
+annotation. If you wish to run some specs/features before others, assign them a lower (negative) run order. If you want
+to run them after the default-ordered elements, assign them a higher (positive) order number:
+
+[source,groovy]
+----
+@Order(1)  // Execute after default-ordered specs
+class FirstSpec extends Specification {
+  // Execute features in order 'three', 'one', 'two' in ascending order of assigned @Order values
+  @Order(2) def one() { expect: true }
+  @Order(3) def two() { expect: true }
+  @Order(1) def three() { expect: true }
+}
+
+@Order(-1)  // Execute before default-ordered specs
+class SecondSpec extends Specification {
+  def foo() { expect: true }                          // Default order
+  @Order(99) def bar() { expect: true }               // Execute after default-ordered features
+  @Order(-5) def zot() { expect: true }               // Execute before default-ordered features
+}
+
+// Default order
+class ThirdSpec extends Specification {
+  def "some feature"() { expect: true }               // Default order
+  @Order(1) def "another feature"() { expect: true }  // Execute after default-ordered features
+  def "one more feature"() { expect: true }           // Default order
+}
+
+// Default order
+class FourthSpec extends Specification {
+  def 'feature X'() { expect: true }                  // Default order
+  def 'feature M'() { expect: true }                  // Default order
+  @Order(-1) def 'feature D'() { expect: true }       // Execute before default-ordered features
+}
+----
+
 == Third-Party Extensions
 
 You can find a list of third-party extensions in the https://github.com/spockframework/spock/wiki/Third-Party-Extensions[Spock Wiki].
@@ -733,13 +853,17 @@ fully-qualified class name in a file `META-INF/services/org.spockframework.runti
 class path. As soon as these two conditions are satisfied, the extension is automatically loaded and used when Spock is
 running.
 
-`IGlobalExtension` has the following three methods:
+`IGlobalExtension` has the following four methods:
 
 `start()`::
   This is called once at the very start of the Spock execution.
 
+`initSpecs(Collection<SpecInfo> specs)`::
+  This is called once, before visiting single specifications later on in `visitSpec`. It enables global extensions to
+  view all specifications as an ensemble, e.g. for iterating over them and rearranging their execution order.
+
 `visitSpec(SpecInfo spec)`::
-  This is called once for each specification. In this method you can prepare a specification with your extension magic,
+  This is called once for each specification. In this method, you can prepare a specification with your extension magic,
   like attaching interceptors to various interception points as described in the chapter <<Interceptors>>.
 
 `stop()`::

docs/release_notes.adoc

@@ -5,6 +5,12 @@ include::include.adoc[]
 
 == 2.4 (tbd)
 
+* New lifecycle method `IGlobalExtension.initSpecs` spockPull:1631[]
+* Support for setting execution order (a.k.a. run order) in `SpecInfo`, can be used by extensions spockPull:1631[]
+* New built-in `OrderExtension` supports run order modification for specifications, features or a combination of both.
+  Built-in orderers can create random, alphabetical or user-defined ordering, the latter using the new `@Order(int)`
+  annotation. See manual section <<extensions#_run_order,Run Order>>. spockPull:1631[]
+
 == 2.4-M1 (2022-11-30)
 
 * Fix issues with Spring 6/Spring Boot 3 spockPull:1541[]

spock-core/src/main/java/org/spockframework/runtime/ExtensionRunner.java

@@ -14,6 +14,7 @@
 
 package org.spockframework.runtime;
 
+import org.junit.platform.engine.TestDescriptor;
 import org.spockframework.runtime.extension.*;
 import org.spockframework.runtime.model.*;
 import org.spockframework.util.Nullable;
@@ -30,6 +31,8 @@
  */
 @SuppressWarnings("rawtypes")
 public class ExtensionRunner {
+  private static final SpecInfo EMPTY_SPEC = new SpecInfo();
+
   private final SpecInfo spec;
   private final IExtensionRegistry extensionRegistry;
   private final IConfigurationRegistry configurationRegistry;
@@ -42,6 +45,20 @@ public ExtensionRunner(SpecInfo spec, IExtensionRegistry extensionRegistry, ICon
     this.configurationRegistry = configurationRegistry;
   }
 
+  public ExtensionRunner(IExtensionRegistry extensionRegistry, IConfigurationRegistry configurationRegistry) {
+    this(EMPTY_SPEC, extensionRegistry, configurationRegistry);
+  }
+
+  public void initGlobalExtensions(Set<? extends TestDescriptor> testDescriptors) {
+    List<SpecInfo> specs = testDescriptors.stream()
+      .filter(testDescriptor -> testDescriptor instanceof SpecNode)
+      .map(testDescriptor -> ((SpecNode) testDescriptor).getNodeInfo())
+      .collect(toList());
+    for (IGlobalExtension extension : extensionRegistry.getGlobalExtensions()) {
+      extension.initSpecs(specs);
+    }
+  }
+
   public void run() {
     runGlobalExtensions();
     runAnnotationDrivenExtensions();

spock-core/src/main/java/org/spockframework/runtime/RunContext.java

@@ -81,6 +81,10 @@ public ExtensionRunner createExtensionRunner(SpecInfo spec) {
     return new ExtensionRunner(spec, globalExtensionRegistry, globalExtensionRegistry);
   }
 
+  public ExtensionRunner createExtensionRunner() {
+    return new ExtensionRunner(globalExtensionRegistry, globalExtensionRegistry);
+  }
+
   public PlatformParameterizedSpecRunner createSpecRunner(SpecInfo spec) {
     return new PlatformParameterizedSpecRunner(
       new MasterRunSupervisor(spec, createStackTraceFilter(spec), diffedObjectRenderer));

spock-core/src/main/java/org/spockframework/runtime/SpecProcessor.java

@@ -0,0 +1,35 @@
+/*
+ * Copyright 2023 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.spockframework.runtime;
+
+import org.spockframework.runtime.extension.builtin.orderer.SpecOrderer;
+import org.spockframework.runtime.model.SpecInfo;
+
+import java.util.Collection;
+
+/**
+ * Generic bulk processor for a collection of {@link SpecInfo} elements
+ *
+ * @see SpecOrderer
+ */
+public interface SpecProcessor {
+  /**
+   * Bulk-process a collection of {@link SpecInfo} elements in-place, i.e. do not return anything but operate on the
+   * elements given, changing their state if necessary.
+   *
+   * @param specs spec-info instances to be processed
+   */
+  void process(Collection<SpecInfo> specs);
+}

spock-core/src/main/java/org/spockframework/runtime/SpockEngineDiscoveryPostProcessor.java

@@ -5,15 +5,22 @@
 
 import org.junit.platform.engine.*;
 
+import java.util.Set;
+
+import static java.util.Comparator.comparingInt;
+
 class SpockEngineDiscoveryPostProcessor {
 
   private static final Object[] EMPTY_ARGS = new Object[0];
 
   SpockEngineDescriptor postProcessEngineDescriptor(UniqueId uniqueId, RunContext runContext,
     SpockEngineDescriptor engineDescriptor) {
     SpockEngineDescriptor processedEngineDescriptor = new SpockEngineDescriptor(uniqueId, runContext);
-    engineDescriptor.getChildren().stream()
+    Set<? extends TestDescriptor> testDescriptors = engineDescriptor.getChildren();
+    initSpecNodes(testDescriptors, runContext);
+    testDescriptors.stream()
       .map(child -> processSpecNode(child, runContext))
+      .sorted(comparingInt(child -> child instanceof SpecNode ? ((SpecNode) child).getNodeInfo().getExecutionOrder() : 0))
       .forEach(processedEngineDescriptor::addChild);
     return processedEngineDescriptor;
   }
@@ -43,6 +50,10 @@ private UniqueId toUniqueId(UniqueId parentId, FeatureInfo feature) {
     return parentId.append("feature", feature.getFeatureMethod().getReflection().getName());
   }
 
+  private void initSpecNodes(Set<? extends TestDescriptor> testDescriptors, RunContext runContext) {
+    runContext.createExtensionRunner().initGlobalExtensions(testDescriptors);
+  }
+
   private TestDescriptor processSpecNode(TestDescriptor child, RunContext runContext) {
     if (child instanceof SpecNode) {
       SpecNode specNode = (SpecNode) child;

spock-core/src/main/java/org/spockframework/runtime/extension/IGlobalExtension.java

@@ -18,10 +18,13 @@
 
 import org.spockframework.runtime.model.SpecInfo;
 
+import java.util.Collection;
+
 // TODO: start/stop lifecycle
 // TODO: design threading model
 public interface IGlobalExtension {
-  default void start() {};
-  default void visitSpec(SpecInfo spec){};
-  default void stop(){};
+  default void start() {}
+  default void initSpecs(Collection<SpecInfo> specs) {}
+  default void visitSpec(SpecInfo spec) {}
+  default void stop() {}
 }

spock-core/src/main/java/org/spockframework/runtime/extension/builtin/OrderExtension.java

@@ -0,0 +1,34 @@
+/*
+ * Copyright 2023 the original author or authors.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.spockframework.runtime.extension.builtin;
+
+import org.spockframework.runtime.extension.IGlobalExtension;
+import org.spockframework.runtime.model.SpecInfo;
+import spock.config.RunnerConfiguration;
+
+import java.util.Collection;
+
+public class OrderExtension implements IGlobalExtension {
+  private final RunnerConfiguration config;
+
+  public OrderExtension(RunnerConfiguration config) {
+    this.config = config;
+  }
+
+  @Override
+  public void initSpecs(Collection<SpecInfo> specs) {
+    config.orderer.process(specs);
+  }
+}