Server side AB testing

Sometimes the scope of a test goes beyond what can be achieved in a client-side test, e.g. when it would hurt performance so much that results would be skewed. In these cases, server-side AB tests are your guys!

The infrastructure is a knock-off of the one available in frontend, only much simplified.

Advantages and disadvantages

This is straight out of the frontend book:

Advantages

• Allows you to run tests that just won't work in javascript ie if you want to render a whole different page
• Avoid javascript rendering problems (eg a jump if a DOM element is rendered and then hidden)

Disadvantages

• Can only do AB tests, not multivariant ABC tests
• Only one test can use a given bucket at a time eg. if someone is running a 50:50 test, you can't run a 50:50 until they finish (current setup)
• Slightly harder to set up, preview and read test results

Configuration

In order to set up a test, there is only one thing you need to do: update the configuration with the description of your test! It looks like this:

{
  name=newPaymentFlow
  description=Redesign of the payment flow UI
  segment=Perc0
  state=On
}

and needs to be added inside the abtests configuration array. A test can be enabled/disabled by setting the state key either to On or Off.

Then depending on the pool of users you want to "recruit" for your test, you will specify a different segment. At the moment, only two segments are supported: Perc0 for 0% tests, and Perc50 for 50% tests. The former is for features in development that need to be released so that people in the organisation can see them in situ before they become available to all.

Manual opt-in

1. install the browser extension Header Hacker
2. click on the Header Hacker icon in the browser tool bar and then Settings
3. navigate to Custom Request Headers
   • in the Header field enter the value X-GU-Experiment
   • in the String field enter the variant you want to be in c.f. Valid Variants
4. navigate to Permanent Header Switches
   • in the Domain field enter the value support.theguardian.com
   • in the Header Switch field select the custom request header you created in step 3

Valid Variants

• Perc0-control
• Perc0-variant
• Perc50-control
• Perc50-variant