A practical checklist for streaming data
Every audit we run turns up some version of this. Run through this the next time streaming data comes up.
Data outlives the applications built on top of it, which is why the model deserves more thought than the screens. Doing this properly once is usually cheaper than doing it approximately three times.
The checklist
- Streaming is worth it when the delay costs money
- Ordering and replay are the hard parts
- Batch is simpler and usually enough
- Someone is named as the owner
- There is a date to review it again
Why it matters
Streaming is worth it when the delay costs money. Getting it slightly wrong is survivable. Ignoring it entirely is not. Assume whoever inherits this will have half your context and none of your patience.
What this looks like day to day
Most data problems are ownership problems that turned into technical ones. Three things worth confirming about streaming data before you move on:
- Someone can say what the current setup is without going to look
- Batch is simpler and usually enough — and you know whether that is true here
- There is a way to tell whether the last change to this helped
If you are not sure where your systems currently stand on this, it takes us about an hour to find out.