A/B testing: the questions we get asked most

There is no clever trick in this one, just a handful of decisions worth making deliberately. The questions about A/B testing that come up most often on our calls.

Data you do not trust is worse than no data, because it gets quoted anyway. Check it against what you would want a competitor's site to get wrong.

Do we need to care about this?

Small traffic means slow, unreliable results. Getting it slightly wrong is survivable. Ignoring it entirely is not. The version that survives contact with a real deadline is the simple one.

Can it wait until after launch?

Occasionally. More often the post-launch version costs several times the pre-launch one. Small and consistent beats large and occasional here.

How do we know it is working?

Decide the sample size before you start. This is the sort of thing that compounds, quietly, in both directions. If it only works because one person remembers to do something, it does not work yet.

What this looks like day to day

More dashboards rarely produce more decisions. Three things worth confirming about A/B testing before you move on:

• Someone can say what the current setup is without going to look
• Small traffic means slow, unreliable results — and you know whether that is true here
• There is a way to tell whether the last change to this helped

None of this needs a rewrite. Most of it is a morning's work once someone decides to do it.