Signs it is time to revisit A/B testing

There is no clever trick in this one, just a handful of decisions worth making deliberately. A few signals that A/B testing is due some attention.

Data you do not trust is worse than no data, because it gets quoted anyway. Write the reasoning down alongside the decision, because the reasoning is what changes first.

The signals

• Nobody can say when it was last reviewed
• The answer depends on who you ask
• Small traffic means slow, unreliable results
• Test one meaningful change, not five cosmetic ones

Where to start

Decide the sample size before you start. None of that requires a large budget, only a decision and someone to own it. It rarely shows up as a line item, which is exactly why it slips.

How to tell if yours is fine

More dashboards rarely produce more decisions. Three things worth confirming about A/B testing before you move on:

• Someone can say what the current setup is without going to look
• Decide the sample size before you start — and you know whether that is true here
• There is a way to tell whether the last change to this helped

Most of the value here comes from doing the first two things, not all of them.