ETL pipelines: what to get right first

The version of this that works is simpler than the version most people imagine. If you only fix one thing about ETL pipelines this quarter, make it the first item below.

Most data problems are ownership problems that turned into technical ones. Doing this properly once is usually cheaper than doing it approximately three times.

Start here

Pipelines fail silently unless you design them not to. That sounds obvious written down. It is still the thing most often skipped. It is worth deciding this deliberately rather than inheriting whatever the last person set up.

Then this

Make every run re-runnable without duplicating data. Small and consistent beats large and occasional here. It is the sort of thing that looks like polish right up until it costs you an enquiry.

Eventually

Validate at the boundary, not three steps later. None of that requires a large budget, only a decision and someone to own it. The practical test is whether someone new to the project could tell, in a minute, that it had been handled.

What this looks like day to day

Numbers get quoted in meetings long after anyone remembers how they were calculated. Three things worth confirming about ETL pipelines before you move on:

• Someone can say what the current setup is without going to look
• Make every run re-runnable without duplicating data — and you know whether that is true here
• There is a way to tell whether the last change to this helped

Worth checking on your own setup before it becomes someone else's problem to fix.