Server side AB testing

Sometimes the scope of a test goes beyond what can be achieved in a client-side test, e.g. when it would hurt performance so much that results would be skewed. In these cases, server-side AB tests are your guys!

The infrastructure is a knock-off of the one available in frontend, only much simplified.

Advantages and disadvantages

This is straight out of the frontend book:

Advantages

• Allows you to run tests that just won't work in javascript ie if you want to render a whole different page
• Avoid javascript rendering problems (eg a jump if a DOM element is rendered and then hidden)

Disadvantages

• Can only do AB tests, not multivariant ABC tests
• Only one test can use a given bucket at a time eg. if someone is running a 50:50 test, you can't run a 50:50 until they finish (current setup)
• Slightly harder to set up, preview and read test results

Configuration

In order to set up a test, there is only one thing you need to do: update the configuration with the description of your test! It looks like this:

{
  name=newPaymentFlow
  description=Redesign of the payment flow UI
  segment=Perc0
  state=On
}

and needs to be added inside the abtests configuration array. A test can be enabled/disabled by setting the state key either to On or Off.

Then depending on the pool of users you want to "recruit" for your test, you will specify a different segment. At the moment, only two segments are supported: Perc0 for 0% tests, and Perc50 for 50% tests. The former is for features in development that need to be released so that people in the organisation can see them in situ before they become available to all.

Manual opt-in

You can put yourself in a particular bucket by adding an HTTP header to every requests to support. The name of the header is X-GU-Experiment and must contain a value of the form <segment>-<group>, where group can either be control or variant: for example, Perc50-variant.

Use a browser extension such as Header Hacker to add the header.