ETL pipelines: the questions we get asked most

Most teams know this matters. Fewer have decided who owns it. The questions about ETL pipelines that come up most often on our calls.

Most data problems are ownership problems that turned into technical ones. Most teams find the first pass takes an afternoon and the maintenance takes minutes a month.

Do we need to care about this?

Pipelines fail silently unless you design them not to. Where this goes wrong is almost never a lack of knowledge. Assume whoever inherits this will have half your context and none of your patience.

Can it wait until after launch?

Occasionally. More often the post-launch version costs several times the pre-launch one. There is a version of this that is over-engineered, and it is worth avoiding.

How do we know it is working?

Validate at the boundary, not three steps later. That sounds obvious written down. It is still the thing most often skipped. The version that survives contact with a real deadline is the simple one.

How to tell if yours is fine

Numbers get quoted in meetings long after anyone remembers how they were calculated. Three things worth confirming about ETL pipelines before you move on:

• Someone can say what the current setup is without going to look
• Validate at the boundary, not three steps later — and you know whether that is true here
• There is a way to tell whether the last change to this helped

If you are not sure where your systems currently stand on this, it takes us about an hour to find out.