Streaming data, explained without the jargon

The advice here is unglamorous, which is probably why it gets skipped. Here is streaming data without the vocabulary that usually surrounds it.

Data outlives the applications built on top of it, which is why the model deserves more thought than the screens. If two people in the business would answer this differently, that gap is the actual problem.

The short version

Streaming is worth it when the delay costs money. That sounds obvious written down. It is still the thing most often skipped. It rarely shows up as a line item, which is exactly why it slips.

Why people complicate it

Most of the confusion comes from tooling rather than from the idea itself. Where this goes wrong is almost never a lack of knowledge.

Ordering and replay are the hard parts. In practice this is a scheduling problem more than a technical one. Doing this properly once is usually cheaper than doing it approximately three times.

Where to go from here

Batch is simpler and usually enough. In practice this is a scheduling problem more than a technical one. Check it against what you would want a competitor's site to get wrong.

What this looks like day to day

Most data problems are ownership problems that turned into technical ones. Three things worth confirming about streaming data before you move on:

• Someone can say what the current setup is without going to look
• Ordering and replay are the hard parts — and you know whether that is true here
• There is a way to tell whether the last change to this helped

Most of the value here comes from doing the first two things, not all of them.