A practical checklist for ETL pipelines

The gap between knowing this and actually doing it is where most teams lose ground. Run through this the next time ETL pipelines comes up.

Data outlives the applications built on top of it, which is why the model deserves more thought than the screens. It is the sort of thing that looks like polish right up until it costs you an enquiry.

The checklist

• Pipelines fail silently unless you design them not to
• Make every run re-runnable without duplicating data
• Validate at the boundary, not three steps later
• Someone is named as the owner
• There is a date to review it again

What it costs to ignore

Pipelines fail silently unless you design them not to. The reasoning matters more than the rule, because the rule has exceptions. The failure mode is not doing it wrong, it is doing it once and assuming it stays done.

The short version

Numbers get quoted in meetings long after anyone remembers how they were calculated. Three things worth confirming about ETL pipelines before you move on:

• Someone can say what the current setup is without going to look
• Pipelines fail silently unless you design them not to — and you know whether that is true here
• There is a way to tell whether the last change to this helped

If you are not sure where your systems currently stand on this, it takes us about an hour to find out.