What to ask your agency about ETL pipelines
The gap between knowing this and actually doing it is where most teams lose ground. If you are briefing an agency or a freelancer on ETL pipelines, these questions are worth asking early.
Most data problems are ownership problems that turned into technical ones. Most teams find the first pass takes an afternoon and the maintenance takes minutes a month.
Questions worth asking
- Who will actually do this work, and have they done it before?
- How will we know afterwards whether it worked?
- What happens if it needs changing in a year?
- What are you assuming that we have not confirmed?
What a good answer sounds like
Pipelines fail silently unless you design them not to. That sounds obvious written down. It is still the thing most often skipped. Check it against what you would want a competitor's site to get wrong.
Validate at the boundary, not three steps later. Small and consistent beats large and occasional here. Doing this properly once is usually cheaper than doing it approximately three times.
What this looks like day to day
Numbers get quoted in meetings long after anyone remembers how they were calculated. Three things worth confirming about ETL pipelines before you move on:
- Someone can say what the current setup is without going to look
- Make every run re-runnable without duplicating data — and you know whether that is true here
- There is a way to tell whether the last change to this helped
If any of that sounds like a description of your current setup, it is fixable.