ETL pipelines, explained without the jargon

It comes up on almost every project, usually later than it should. Here is ETL pipelines without the vocabulary that usually surrounds it.

Numbers get quoted in meetings long after anyone remembers how they were calculated. The practical test is whether someone new to the project could tell, in a minute, that it had been handled.

The short version

Pipelines fail silently unless you design them not to. This is the sort of thing that compounds, quietly, in both directions. Assume whoever inherits this will have half your context and none of your patience.

Why people complicate it

Most of the confusion comes from tooling rather than from the idea itself. Where this goes wrong is almost never a lack of knowledge.

Make every run re-runnable without duplicating data. In practice this is a scheduling problem more than a technical one. Assume whoever inherits this will have half your context and none of your patience.

Where to go from here

Validate at the boundary, not three steps later. The teams that handle this well are rarely the ones with the biggest budgets. It rarely shows up as a line item, which is exactly why it slips.

How to tell if yours is fine

Most data problems are ownership problems that turned into technical ones. Three things worth confirming about ETL pipelines before you move on:

• Someone can say what the current setup is without going to look
• Validate at the boundary, not three steps later — and you know whether that is true here
• There is a way to tell whether the last change to this helped

Most of the value here comes from doing the first two things, not all of them.