Isolated concurrency utility to run specific tests in isolation

[blake-mealey]

I run my test suite against a shared local DB and I was running into issues with a particular test file which is forced to make some assumptions about the state of the DB while it runs.

I managed to come up with a pretty clean solution for managing this using @ava/cooperate:

// test-concurrency.ts
import type { TestFn } from 'ava';
import { SharedContext } from '@ava/cooperate';

const context = new SharedContext('all-tests');

/**
 * Create a semaphore with a very high initial value. We don't actually want to limit concurrency for most
 * tests but we want a mechanism to force some tests to run in isolation.
 */
const semaphore = context.createSemaphore('all-tests', 1_000_000);

function withSemaphore<TContext>(test: TestFn<TContext>, name: string, amount: number): TestFn<TContext> {
  const test2 = test as TestFn<TContext & { release: () => void }>;
  test2.serial.before(name, async (t) => {
    t.context.release = await semaphore.acquire(amount);
  });
  test2.after.always(name, async (t) => {
    t.context.release();
  });
  // Don't expose `context.release` to tests
  return test2 as TestFn<TContext>;
}

/**
 * Requires an ava test context to down a semaphore shared by all tests by 1. Since the semaphore's initial
 * value is very high this should not affect concurrency for "shared" tests unless an "isolated" test is
 * executing which downs the semaphore's full initial value (see `withIsolatedConcurrency`).
 * @example const test = withSharedConcurrency(ava)
 * @param test - ava test function
 * @returns updated ava test function
 */
export function withSharedConcurrency<TContext>(test: TestFn<TContext>): TestFn<TContext> {
  return withSemaphore(test, 'withShared', 1);
}

/**
 * Requires an ava test context to down a semaphore shared by all tests by its full initial value. This will
 * grant the test context exclusive access until it completes.
 * @example const test = withIsolatedConcurrency(ava)
 * @param test - ava test function
 * @returns updated ava test function
 */
export function withIsolatedConcurrency<TContext>(test: TestFn<TContext>): TestFn<TContext> {
  return withSemaphore(test, 'withIsolated', semaphore.initialValue);
}

Example usage:

// shared-concurrency.test.ts
import ava from 'ava';
import { withSharedConcurrency } from './test-concurrency'

const test = withSharedConcurrency(ava);

test('should run in parallel', (t) => {
  t.pass();
});

// isolated-concurrency.test.ts
import ava from 'ava';
import { withIsolatedConcurrency } from './test-concurrency'

const test = withIsolatedConcurrency(ava);

test('should run in isolation', (t) => {
  t.pass();
});

[novemberborn]

Nice one @blake-mealey, thanks for sharing!

[blake-mealey]

Since posting this, I've been forced to find an alternative to this. I wasn't able to track down the root cause but as we added more tests to our suite, we started seeing random flakiness where some tests would get stuck waiting for the semaphore to down indefinitely.

We ended up finding a different solution for our specific problem which is to create and migrate a test-file-specific database within the DB server (CREATE DATABASE ...) and use that to run the isolated tests against.

This has been working super well without any flakiness and is also faster since the communication between workers was increasing test latency.

[novemberborn]

Yes that's the approach I've taken previously with (emulated) Firestore databases. In fact each test file got its own database. (Easy when you don't need to create tables and manage schemas.)