ETL pipelines for small teams

The version of this that works is simpler than the version most people imagine. Most advice about ETL pipelines assumes a team that does not exist at your size. Here is the version that does not.

Most data problems are ownership problems that turned into technical ones. If two people in the business would answer this differently, that gap is the actual problem.

What to keep

Pipelines fail silently unless you design them not to. In practice this is a scheduling problem more than a technical one. Doing this properly once is usually cheaper than doing it approximately three times.

What to drop

Process that exists to coordinate ten people is overhead when there are two of you. The cost of getting this wrong is rarely visible on the day it happens.

What good looks like

Validate at the boundary, not three steps later. The reasoning matters more than the rule, because the rule has exceptions. Assume whoever inherits this will have half your context and none of your patience.

How to tell if yours is fine

Data outlives the applications built on top of it, which is why the model deserves more thought than the screens. Three things worth confirming about ETL pipelines before you move on:

• Someone can say what the current setup is without going to look
• Make every run re-runnable without duplicating data — and you know whether that is true here
• There is a way to tell whether the last change to this helped

If any of that sounds like a description of your current setup, it is fixable.