A short guide to A/B testing

We end up explaining this on discovery calls often enough that it deserved writing down. Everything we would tell a client about A/B testing in the time it takes to drink a coffee.

Data you do not trust is worse than no data, because it gets quoted anyway. Anything you cannot measure here, you are deciding by taste, which is fine as long as everyone knows it.

The reason this keeps coming up

Small traffic means slow, unreliable results. Small and consistent beats large and occasional here. It rarely shows up as a line item, which is exactly why it slips.

How we handle it

Test one meaningful change, not five cosmetic ones. Getting it slightly wrong is survivable. Ignoring it entirely is not. It is worth deciding this deliberately rather than inheriting whatever the last person set up.

What to watch for

Decide the sample size before you start. Getting it slightly wrong is survivable. Ignoring it entirely is not. Check it against what you would want a competitor's site to get wrong.

In practice

More dashboards rarely produce more decisions. Three things worth confirming about A/B testing before you move on:

• Someone can say what the current setup is without going to look
• Decide the sample size before you start — and you know whether that is true here
• There is a way to tell whether the last change to this helped

The point is not perfection, it is knowing which of these you have consciously chosen to skip.