When observability is worth the effort
This is one of those topics that looks small until it costs you something. Observability is not free, and pretending otherwise leads to bad decisions.
Operability is a feature, and it has to be built rather than bought. The teams that stay on top of it are the ones who put it on a calendar rather than a wish list.
When it is worth it
Monitoring tells you something broke, observability tells you why. It is worth being explicit about, because assumptions differ quietly. It is worth deciding this deliberately rather than inheriting whatever the last person set up.
When it is not
If nothing downstream depends on it and nobody is complaining, it can wait. It is worth being explicit about, because assumptions differ quietly.
How to decide
Instrument the paths that lose money first. That sounds obvious written down. It is still the thing most often skipped. The version that survives contact with a real deadline is the simple one.
How to tell if yours is fine
The bill is a design document: it tells you exactly what your architecture actually does. Three things worth confirming about observability before you move on:
- Someone can say what the current setup is without going to look
- Instrument the paths that lose money first — and you know whether that is true here
- There is a way to tell whether the last change to this helped
If you are not sure where your systems currently stand on this, it takes us about an hour to find out.