A/B testing, explained without the jargon

It comes up on almost every project, usually later than it should. Here is A/B testing without the vocabulary that usually surrounds it.

More dashboards rarely produce more decisions. The failure mode is not doing it wrong, it is doing it once and assuming it stays done.

The short version

Small traffic means slow, unreliable results. This is the sort of thing that compounds, quietly, in both directions. It rarely shows up as a line item, which is exactly why it slips.

Why people complicate it

Most of the confusion comes from tooling rather than from the idea itself. Getting it slightly wrong is survivable. Ignoring it entirely is not.

Test one meaningful change, not five cosmetic ones. That sounds obvious written down. It is still the thing most often skipped. It is worth deciding this deliberately rather than inheriting whatever the last person set up.

What to do next

Decide the sample size before you start. Getting it slightly wrong is survivable. Ignoring it entirely is not. Budget a little time for it every quarter and it never becomes a project of its own.

In practice

Measurement is only useful when someone has agreed in advance what they would do differently. Three things worth confirming about A/B testing before you move on:

• Someone can say what the current setup is without going to look
• Decide the sample size before you start — and you know whether that is true here
• There is a way to tell whether the last change to this helped

Worth checking on your own setup before it becomes someone else's problem to fix.