Three myths about ETL pipelines

Every audit we run turns up some version of this. A few things about ETL pipelines that get repeated more often than they get checked.

Numbers get quoted in meetings long after anyone remembers how they were calculated. It is the sort of thing that looks like polish right up until it costs you an enquiry.

“It only matters for big sites”

Pipelines fail silently unless you design them not to. In practice this is a scheduling problem more than a technical one. It is the sort of thing that looks like polish right up until it costs you an enquiry.

“We can deal with it after launch”

Sometimes true, usually expensive. In practice this is a scheduling problem more than a technical one.

“Our platform handles it”

Validate at the boundary, not three steps later. Getting it slightly wrong is survivable. Ignoring it entirely is not. The practical test is whether someone new to the project could tell, in a minute, that it had been handled.

In practice

Data outlives the applications built on top of it, which is why the model deserves more thought than the screens. Three things worth confirming about ETL pipelines before you move on:

• Someone can say what the current setup is without going to look
• Make every run re-runnable without duplicating data — and you know whether that is true here
• There is a way to tell whether the last change to this helped

If any of that sounds like a description of your current setup, it is fixable.