Five mistakes teams make with streaming data

There is no clever trick in this one, just a handful of decisions worth making deliberately. These are the ones we run into repeatedly when we audit streaming data.

Most data problems are ownership problems that turned into technical ones. It is worth deciding this deliberately rather than inheriting whatever the last person set up.

Common failure modes

• Treating it as a launch task rather than an ongoing one
• Assuming someone else already owns it
• Streaming is worth it when the delay costs money
• Ordering and replay are the hard parts
• Never checking whether the fix actually worked

Batch is simpler and usually enough. Where this goes wrong is almost never a lack of knowledge. It rarely shows up as a line item, which is exactly why it slips.

What to do next

How to tell if yours is fine

Numbers get quoted in meetings long after anyone remembers how they were calculated. Three things worth confirming about streaming data before you move on:

• Someone can say what the current setup is without going to look
• Streaming is worth it when the delay costs money — and you know whether that is true here
• There is a way to tell whether the last change to this helped

None of this needs a rewrite. Most of it is a morning's work once someone decides to do it.