Three myths about streaming data

This is cheap to get right at the start and expensive to retrofit. A few things about streaming data that get repeated more often than they get checked.

Most data problems are ownership problems that turned into technical ones. It is worth deciding this deliberately rather than inheriting whatever the last person set up.

“It only matters for big sites”

Streaming is worth it when the delay costs money. That sounds obvious written down. It is still the thing most often skipped. The practical test is whether someone new to the project could tell, in a minute, that it had been handled.

“We can deal with it after launch”

Sometimes true, usually expensive. The reasoning matters more than the rule, because the rule has exceptions.

“Our platform handles it”

Batch is simpler and usually enough. Where this goes wrong is almost never a lack of knowledge. Most teams find the first pass takes an afternoon and the maintenance takes minutes a month.

The short version

Numbers get quoted in meetings long after anyone remembers how they were calculated. Three things worth confirming about streaming data before you move on:

• Someone can say what the current setup is without going to look
• Streaming is worth it when the delay costs money — and you know whether that is true here
• There is a way to tell whether the last change to this helped

Most of the value here comes from doing the first two things, not all of them.