Three myths about A/B testing

The version of this that works is simpler than the version most people imagine. A few things about A/B testing that get repeated more often than they get checked.

Data you do not trust is worse than no data, because it gets quoted anyway. The teams that stay on top of it are the ones who put it on a calendar rather than a wish list.

“It only matters for big sites”

Small traffic means slow, unreliable results. The reasoning matters more than the rule, because the rule has exceptions. Anything you cannot measure here, you are deciding by taste, which is fine as long as everyone knows it.

“We can deal with it after launch”

Sometimes true, usually expensive. The cost of getting this wrong is rarely visible on the day it happens.

“Our platform handles it”

Decide the sample size before you start. There is a version of this that is over-engineered, and it is worth avoiding. The practical test is whether someone new to the project could tell, in a minute, that it had been handled.

The short version

Measurement is only useful when someone has agreed in advance what they would do differently. Three things worth confirming about A/B testing before you move on:

• Someone can say what the current setup is without going to look
• Decide the sample size before you start — and you know whether that is true here
• There is a way to tell whether the last change to this helped

None of this needs a rewrite. Most of it is a morning's work once someone decides to do it.