Five mistakes teams make with ETL pipelines

There is no clever trick in this one, just a handful of decisions worth making deliberately. These are the ones we run into repeatedly when we audit ETL pipelines.

Numbers get quoted in meetings long after anyone remembers how they were calculated. The failure mode is not doing it wrong, it is doing it once and assuming it stays done.

Common failure modes

• Treating it as a launch task rather than an ongoing one
• Assuming someone else already owns it
• Pipelines fail silently unless you design them not to
• Make every run re-runnable without duplicating data
• Never checking whether the fix actually worked

Validate at the boundary, not three steps later. In practice this is a scheduling problem more than a technical one. It rarely shows up as a line item, which is exactly why it slips.

Making it stick

How to tell if yours is fine

Data outlives the applications built on top of it, which is why the model deserves more thought than the screens. Three things worth confirming about ETL pipelines before you move on:

• Someone can say what the current setup is without going to look
• Validate at the boundary, not three steps later — and you know whether that is true here
• There is a way to tell whether the last change to this helped

Worth checking on your own setup before it becomes someone else's problem to fix.