Before you invest in streaming data

Teams tend to reach for this after something has already gone wrong. Before you spend anything on streaming data, it is worth confirming a few things are already true.

Most data problems are ownership problems that turned into technical ones. Anything you cannot measure here, you are deciding by taste, which is fine as long as everyone knows it.

Prerequisites

• You can describe the outcome you want in one sentence
• Someone owns it after the work is done
• Streaming is worth it when the delay costs money
• You have a way to tell whether it worked

The mistakes we see most

Ordering and replay are the hard parts. Where this goes wrong is almost never a lack of knowledge. Doing this properly once is usually cheaper than doing it approximately three times.

Batch is simpler and usually enough. It is worth being explicit about, because assumptions differ quietly. Check it against what you would want a competitor's site to get wrong.

What this looks like day to day

Numbers get quoted in meetings long after anyone remembers how they were calculated. Three things worth confirming about streaming data before you move on:

• Someone can say what the current setup is without going to look
• Batch is simpler and usually enough — and you know whether that is true here
• There is a way to tell whether the last change to this helped

Worth checking on your own setup before it becomes someone else's problem to fix.