A/B testing: a practical guide
This is one of those topics that looks small until it costs you something. This guide covers what A/B testing actually involves, where it usually goes wrong, and how to tell whether yours is in reasonable shape.
Measurement is only useful when someone has agreed in advance what they would do differently. Nothing below assumes a large team or a large budget — most of it is a decision somebody has to make and then write down.
Why it matters
Small traffic means slow, unreliable results. The cost of getting this wrong is rarely visible on the day it happens. If two people in the business would answer this differently, that gap is the actual problem.
For most businesses the question is not whether this matters but how much of it is worth doing right now. That depends on what you are trying to achieve in the next few months, not on best practice in the abstract. None of that requires a large budget, only a decision and someone to own it.
The practical version
Test one meaningful change, not five cosmetic ones. The teams that handle this well are rarely the ones with the biggest budgets. Assume whoever inherits this will have half your context and none of your patience.
Data you do not trust is worse than no data, because it gets quoted anyway. The version that works in practice is usually less elaborate than the version described in the guides.
Decide the sample size before you start. Small and consistent beats large and occasional here. The failure mode is not doing it wrong, it is doing it once and assuming it stays done.
A working checklist
If you want a quick read on where you stand, work through this. Anything you cannot answer confidently is where to start.
- Small traffic means slow, unreliable results
- Test one meaningful change, not five cosmetic ones
- Decide the sample size before you start
- Someone is named as the owner, not just assumed to be
- There is a date in the calendar to review it again
- The decision and the reasoning behind it are written down somewhere findable
- You could explain the current setup to a new hire in five minutes
The mistakes we see most
The most common failure is not doing this badly. It is doing it once, during a launch, and never revisiting it. Circumstances move, the setup does not, and the gap widens quietly until something breaks or somebody notices the numbers.
- It was configured during a launch and has not been touched since
- Different people in the business believe different things are true about it
- There is no way to tell whether the last change helped or hurt
- The only person who understands it has left, or is about to
More dashboards rarely produce more decisions. The cost of getting this wrong is rarely visible on the day it happens.
How we approach it
On our projects this gets handled during the build rather than added afterwards, because retrofitting it costs several times more than including it. We write down what was decided and why, so the next person to touch it is not guessing.
If you are working with someone else, the questions worth asking are simple: who owns this, how will we know it is working, and what happens when it needs to change?
Making it stick
Pick the single item from the checklist above that would cause the most trouble if it turned out to be wrong. Fix that one, confirm it worked, then move on. None of this needs a rewrite. Most of it is a morning's work once someone decides to do it.