Refactor CI test matrix for clearer coverage and faster execution

[baconYao]

Is there an existing issue for this?

• There is no existing issue for this feature

What are you currently unable to do

I noticed that the System-related tests in Incus’ GitHub Actions workflow are particularly time-consuming. After a brief discussion with Stéphane, he explained that the goal is to improve the overall execution time of the Incus test suite on GitHub Actions. Currently, a full test run takes around 1 hour and 20 minutes, and the target is to reduce this to a more reasonable ~30 minutes.

This can be achieved by splitting the existing test jobs into smaller units and leveraging the parallel execution capabilities of GitHub Actions runners, thereby reducing the runtime of each individual job.

Problem Analysis

• Currently, most system-related tests take between 10 and 40 minutes, which is still within a reasonable range. However, there is a system test using Ceph as the backend that consumes approximately 1 hour and 20 minutes.

  • https://github.com/lxc/incus/actions/runs/19128144240/job/54706766464
  • https://github.com/lxc/incus/actions/runs/19003812371/job/54279994703

• At present, all tests are only roughly divided into two suites: cluster and standalone. This causes each GitHub Actions job to handle too many tests, resulting in extended execution times.

Improvement Approach

1. Split test jobs into finer-grained units

• Refine existing suite structure
  • Building upon the original architecture with all, standalone, and cluster as suites, further categorize existing test cases into seven distinct categories (need to confirm what):
    • Benefit: This approach reduces the number of test cases each individual job has to run, thereby lowering the execution time per job and improving overall CI efficiency.
    • Categories: core, storage, network, security, misc, instances

Dimension	Values
suite	cluster, standalone
test-category	core, storage, network, security, misc, instances
backend	dir, btrfs, lvm, zfs, ceph, linstor, random
go	oldstable, stable, tip
os	ubuntu-24.04, ubuntu-24.04-arm

2. Reassessing the Test Matrix

After introducing the concept of test-category in Step 1 to split tests into different groups, the side effect is a combinatorial explosion of job combinations. Although GitHub Actions provides parallel runners, handling a large number of jobs at once remains challenging.

Therefore, I believe it is necessary to carefully review and optimize the test grouping. It will reduce the number of jobs to avoid long queue times for individual jobs, which can extend the overall test workflow duration.

[baconYao]

Below are the experimental results after reorganizing the test cases for both the standalone and cluster suites into separate groups.

Step 1. Experiments after landing test-category concept

• Experiment 1
  • Overall Usage: https://github.com/baconYao/incus/actions/runs/19113852099/usage
  • Total Duration: 1 h 9 m
  • Jobs exceeding 20 min:
    • System (stable, standalone, misc, ceph, ubuntu-24.04) – 33 m 6 s
    • System (stable, standalone, misc, linstor, ubuntu-24.04) – 22 m 28 s
    • System (stable, standalone, storage, ceph, ubuntu-24.04) – 22 m 1 s
• Experiment 2
  • Overall Usage: https://github.com/baconYao/incus/actions/runs/19138493765/usage
  • Total Duration: 49 m
  • Jobs exceeding 20 min:
    • System (stable, standalone, storage, zfs, ubuntu-24.04) – 28 m 53 s
    • System (stable, standalone, misc, ceph, ubuntu-24.04) – 22 m 22 s
    • System (stable, cluster, storage, linstor, ubuntu-24.04) – 22 m 8 s
    • System (stable, standalone, storage, ceph, ubuntu-24.04) – 21 m 35 s

Result

• Before the Change
  • Total Jobs: 34 (Document + Client + Code + System = 1 + 6 + 3 + 24)
  • Longest-running Job:
  • System (stable, standalone, ceph, ubuntu-24.04) – 1 h 12 m 14 s
  • Overall Usage: Link
• After the Change
  • Total Jobs: 90 (Document + Client + Code + System = 1 + 6 + 3 + 80)
  • Conclusion:
    • Improvement:
      • Each Job now has fewer test items to execute, reducing the runtime per Job.
    • Drawback:
      • The total number of Jobs more than doubled (34 → 90).
      • GitHub Actions can run up to 20 jobs parallelly, so excess jobs may spend time in the queue, limiting the reduction in overall workflow execution time. Therefore, the total duration observed in Experiment 1 and Experiment 2 fluctuates, as the execution order of jobs can amplify and impact the overall workflow runtime.

Step 2. Adjusting Test Combinations

Based on observations, the choice of backend has a significant impact on test duration. Therefore, I propose using the dir backend for running the full suite of tests across all categories. For backend-specific tests, we will focus only on the storage category within cluster and standalone suites. This approach balances comprehensive coverage with efficient test execution.

Result

• Experiment 1
  • Overall Usage: https://github.com/baconYao/incus/actions/runs/19141772084/attempts/1
  • Workflow Duration: 32 m 7 s
  • Jobs exceeding 20 minutes:
    • System (stable, standalone, storage, lvm, ubuntu-24.04) – 26 m 1 s
    • System (stable, cluster, storage, linstor, ubuntu-24.04) – 23 m 56 s
    • System (stable, standalone, storage, ceph, ubuntu-24.04) – 21 m 24 s

Conclusion

By adopting the above approach to adjust Incus’s Github Actions, the overall workflow test duration can be significantly reduced from approximately 1 hour and 20 minutes to around 32 minutes. Additionally, by categorizing the test cases, we improve maintainability, readability, and scalability, while also enabling more targeted testing for specific categories.

[baconYao]

Feedback from @stgraber

It's great to see the improvements when running with more CI jobs.
The LXC org has a much higher concurrency limit than 20, we have at least 60 if not even higher due to being part of some open source maintainer program with Github.

Looking at the timing of the tests, I think it only really makes sense to split the standalone tests. The cluster tests already complete in less than 30 minutes even for the slow ones.

The split you have for standalone seems reasonable though, we see each of the split taking a decent amount of time to run, making the extra setup time be worth it.