Five mistakes teams make with A/B testing

There is no clever trick in this one, just a handful of decisions worth making deliberately. These are the ones we run into repeatedly when we audit A/B testing.

Measurement is only useful when someone has agreed in advance what they would do differently. It rarely shows up as a line item, which is exactly why it slips.

What to watch for

• Treating it as a launch task rather than an ongoing one
• Assuming someone else already owns it
• Small traffic means slow, unreliable results
• Test one meaningful change, not five cosmetic ones
• Never checking whether the fix actually worked

Decide the sample size before you start. Where this goes wrong is almost never a lack of knowledge. It is the sort of thing that looks like polish right up until it costs you an enquiry.

What to do next

How to tell if yours is fine

More dashboards rarely produce more decisions. Three things worth confirming about A/B testing before you move on:

• Someone can say what the current setup is without going to look
• Test one meaningful change, not five cosmetic ones — and you know whether that is true here
• There is a way to tell whether the last change to this helped

The point is not perfection, it is knowing which of these you have consciously chosen to skip.