A short guide to streaming data

Most teams know this matters. Fewer have decided who owns it. Everything we would tell a client about streaming data in the time it takes to drink a coffee.

Most data problems are ownership problems that turned into technical ones. Most teams find the first pass takes an afternoon and the maintenance takes minutes a month.

What is actually at stake

Streaming is worth it when the delay costs money. Where this goes wrong is almost never a lack of knowledge. Write the reasoning down alongside the decision, because the reasoning is what changes first.

What good looks like

Ordering and replay are the hard parts. The cost of getting this wrong is rarely visible on the day it happens. The failure mode is not doing it wrong, it is doing it once and assuming it stays done.

Where it usually goes wrong

Batch is simpler and usually enough. The cost of getting this wrong is rarely visible on the day it happens. The version that survives contact with a real deadline is the simple one.

In practice

Data outlives the applications built on top of it, which is why the model deserves more thought than the screens. Three things worth confirming about streaming data before you move on:

• Someone can say what the current setup is without going to look
• Ordering and replay are the hard parts — and you know whether that is true here
• There is a way to tell whether the last change to this helped

Pick the one that would hurt most if it failed, and start there.