A practical checklist for A/B testing

It comes up on almost every project, usually later than it should. Run through this the next time A/B testing comes up.

Measurement is only useful when someone has agreed in advance what they would do differently. If it only works because one person remembers to do something, it does not work yet.

The checklist

• Small traffic means slow, unreliable results
• Test one meaningful change, not five cosmetic ones
• Decide the sample size before you start
• Someone is named as the owner
• There is a date to review it again

The reason this keeps coming up

Small traffic means slow, unreliable results. It is worth being explicit about, because assumptions differ quietly. Assume whoever inherits this will have half your context and none of your patience.

How to tell if yours is fine

Data you do not trust is worse than no data, because it gets quoted anyway. Three things worth confirming about A/B testing before you move on:

• Someone can say what the current setup is without going to look
• Test one meaningful change, not five cosmetic ones — and you know whether that is true here
• There is a way to tell whether the last change to this helped

Pick the one that would hurt most if it failed, and start there.