How to get streaming data right
The version of this that works is simpler than the version most people imagine. The short answer to streaming data is that it is mostly a sequence of small decisions, not one big one.
Numbers get quoted in meetings long after anyone remembers how they were calculated. Most teams find the first pass takes an afternoon and the maintenance takes minutes a month.
Why this earns attention
Streaming is worth it when the delay costs money. In practice this is a scheduling problem more than a technical one. It is the sort of thing that looks like polish right up until it costs you an enquiry.
The steps
- Establish what you have today before changing anything
- Ordering and replay are the hard parts
- Batch is simpler and usually enough
- Write down the decision so the next person does not re-litigate it
Batch is simpler and usually enough. Getting it slightly wrong is survivable. Ignoring it entirely is not. It is worth deciding this deliberately rather than inheriting whatever the last person set up.
What to do next
What this looks like day to day
Data outlives the applications built on top of it, which is why the model deserves more thought than the screens. Three things worth confirming about streaming data before you move on:
- Someone can say what the current setup is without going to look
- Streaming is worth it when the delay costs money — and you know whether that is true here
- There is a way to tell whether the last change to this helped
None of this needs a rewrite. Most of it is a morning's work once someone decides to do it.