Before you invest in A/B testing

It comes up on almost every project, usually later than it should. Before you spend anything on A/B testing, it is worth confirming a few things are already true.

Data you do not trust is worse than no data, because it gets quoted anyway. The failure mode is not doing it wrong, it is doing it once and assuming it stays done.

Prerequisites

• You can describe the outcome you want in one sentence
• Someone owns it after the work is done
• Small traffic means slow, unreliable results
• You have a way to tell whether it worked

Where it usually goes wrong

Test one meaningful change, not five cosmetic ones. Where this goes wrong is almost never a lack of knowledge. Most teams find the first pass takes an afternoon and the maintenance takes minutes a month.

Decide the sample size before you start. None of that requires a large budget, only a decision and someone to own it. Check it against what you would want a competitor's site to get wrong.

How to tell if yours is fine

Measurement is only useful when someone has agreed in advance what they would do differently. Three things worth confirming about A/B testing before you move on:

• Someone can say what the current setup is without going to look
• Decide the sample size before you start — and you know whether that is true here
• There is a way to tell whether the last change to this helped

Pick the one that would hurt most if it failed, and start there.