Streaming data: what to get right first

The advice here is unglamorous, which is probably why it gets skipped. If you only fix one thing about streaming data this quarter, make it the first item below.

Most data problems are ownership problems that turned into technical ones. It rarely shows up as a line item, which is exactly why it slips.

Start here

Streaming is worth it when the delay costs money. The reasoning matters more than the rule, because the rule has exceptions. It is worth deciding this deliberately rather than inheriting whatever the last person set up.

Then this

Ordering and replay are the hard parts. Getting it slightly wrong is survivable. Ignoring it entirely is not. It is worth deciding this deliberately rather than inheriting whatever the last person set up.

Eventually

Batch is simpler and usually enough. This is the sort of thing that compounds, quietly, in both directions. It is the sort of thing that looks like polish right up until it costs you an enquiry.

In practice

Data outlives the applications built on top of it, which is why the model deserves more thought than the screens. Three things worth confirming about streaming data before you move on:

• Someone can say what the current setup is without going to look
• Ordering and replay are the hard parts — and you know whether that is true here
• There is a way to tell whether the last change to this helped

The point is not perfection, it is knowing which of these you have consciously chosen to skip.