How to get alerting right
Most teams know this matters. Fewer have decided who owns it. The short answer to alerting is that it is mostly a sequence of small decisions, not one big one.
The bill is a design document: it tells you exactly what your architecture actually does. The practical test is whether someone new to the project could tell, in a minute, that it had been handled.
Why it matters
An alert nobody acts on trains everyone to ignore alerts. Getting it slightly wrong is survivable. Ignoring it entirely is not. It is worth deciding this deliberately rather than inheriting whatever the last person set up.
The steps
- Establish what you have today before changing anything
- Page on symptoms customers feel, not on every anomaly
- Every alert should link to what to do about it
- Write down the decision so the next person does not re-litigate it
Every alert should link to what to do about it. In practice this is a scheduling problem more than a technical one. It is the sort of thing that looks like polish right up until it costs you an enquiry.
What to do next
The short version
Operability is a feature, and it has to be built rather than bought. Three things worth confirming about alerting before you move on:
- Someone can say what the current setup is without going to look
- An alert nobody acts on trains everyone to ignore alerts — and you know whether that is true here
- There is a way to tell whether the last change to this helped
The point is not perfection, it is knowing which of these you have consciously chosen to skip.