How to get A/B testing right
This is one of those topics that looks small until it costs you something. The short answer to A/B testing is that it is mostly a sequence of small decisions, not one big one.
Data you do not trust is worse than no data, because it gets quoted anyway. It rarely shows up as a line item, which is exactly why it slips.
Why it matters
Small traffic means slow, unreliable results. None of that requires a large budget, only a decision and someone to own it. The practical test is whether someone new to the project could tell, in a minute, that it had been handled.
The steps
- Establish what you have today before changing anything
- Test one meaningful change, not five cosmetic ones
- Decide the sample size before you start
- Write down the decision so the next person does not re-litigate it
Decide the sample size before you start. The cost of getting this wrong is rarely visible on the day it happens. Budget a little time for it every quarter and it never becomes a project of its own.
Making it stick
How to tell if yours is fine
Measurement is only useful when someone has agreed in advance what they would do differently. Three things worth confirming about A/B testing before you move on:
- Someone can say what the current setup is without going to look
- Decide the sample size before you start — and you know whether that is true here
- There is a way to tell whether the last change to this helped
The point is not perfection, it is knowing which of these you have consciously chosen to skip.