Why ETL pipelines matters more than it looks
It comes up on almost every project, usually later than it should. ETL pipelines is easy to treat as a detail, and that is exactly why it is worth a few minutes of attention.
Numbers get quoted in meetings long after anyone remembers how they were calculated. The failure mode is not doing it wrong, it is doing it once and assuming it stays done.
The reason this keeps coming up
Pipelines fail silently unless you design them not to. It is worth being explicit about, because assumptions differ quietly. It is worth deciding this deliberately rather than inheriting whatever the last person set up.
Make every run re-runnable without duplicating data. The cost of getting this wrong is rarely visible on the day it happens. If it only works because one person remembers to do something, it does not work yet.
Where it usually goes wrong
Validate at the boundary, not three steps later. There is a version of this that is over-engineered, and it is worth avoiding. Assume whoever inherits this will have half your context and none of your patience.
What this looks like day to day
Most data problems are ownership problems that turned into technical ones. Three things worth confirming about ETL pipelines before you move on:
- Someone can say what the current setup is without going to look
- Validate at the boundary, not three steps later — and you know whether that is true here
- There is a way to tell whether the last change to this helped
None of this needs a rewrite. Most of it is a morning's work once someone decides to do it.