Buildkite Rails Parallel Pipeline Example

This repository is an example Buildkite pipeline that demonstrates how to run 20 parallel test jobs for a Rails application using rbenv and Knapsack.

👉 See this example in action: buildkite/rails-parallel-example

See the full Getting Started Guide for step-by-step instructions on how to get this running, or try it yourself:

How it works

This repository doesn't include a Rails application, but it does provide the CI setup you'd typically use in a real project.

In the pipeline configuration file the parallelism property for the test step is set to 20. When a build, the step will appear 20 times in the pipeline, each with different environment variables exposed so you can divvy up your test suite accordingly. You can then run 20 agents (on the same machine, or distributed) to work on the 20 jobs in parallel.

This example:

• Uses rbenv to manage Ruby versions
• Runs tests in parallel using Buildkite’s parallelism feature
• Leverages Knapsack for test suite splitting
• Can be used on a single machine or distributed across multiple agents

Key files:

• .buildkite/hooks/environment: loads rbenv before steps run
• .buildkite/pipeline.yml: sets up parallel steps
• scripts/ci/setup.sh: prepares the CI environment
• scripts/ci/parallel_specs.sh: runs specs via Knapsack

Parallel test execution

The test step in pipeline.yml is configured like this:

steps:
  - name: ":rspec:"
    command: "scripts/ci/setup.sh && scripts/ci/parallel_specs.sh"
    parallelism: 20

Each parallel job receives unique environment variables (BUILDKITE_PARALLEL_JOB, BUILDKITE_PARALLEL_JOB_COUNT) to help split the workload.

See the parallelizing builds guide for more information to create parallelized and distributed builds with Buildkite.

💡 Want to use Docker? Try the Rails Docker Parallel Example

Knapsack Pro

Consult the Knapsack documentation for configuring your database and dependent services to support running parallel steps on the one machine.

See Knapsack Pro version of this example pipeline for how to perform dynamic splits/allocation of test nodes.

License

See LICENSE.md (MIT)