Getting started with A/B testing
The version of this that works is simpler than the version most people imagine. A short on-ramp to A/B testing for teams who have not touched it before.
Measurement is only useful when someone has agreed in advance what they would do differently. Check it against what you would want a competitor's site to get wrong.
Why this earns attention
Small traffic means slow, unreliable results. None of that requires a large budget, only a decision and someone to own it. It is worth deciding this deliberately rather than inheriting whatever the last person set up.
Your first week
- Find out what is already in place
- Test one meaningful change, not five cosmetic ones
- Change one thing and measure it
Decide the sample size before you start. The cost of getting this wrong is rarely visible on the day it happens. If two people in the business would answer this differently, that gap is the actual problem.
What this looks like day to day
Data you do not trust is worse than no data, because it gets quoted anyway. Three things worth confirming about A/B testing before you move on:
- Someone can say what the current setup is without going to look
- Small traffic means slow, unreliable results — and you know whether that is true here
- There is a way to tell whether the last change to this helped
Pick the one that would hurt most if it failed, and start there.