How to get eventual consistency right
This is one of those topics that looks small until it costs you something. The short answer to eventual consistency is that it is mostly a sequence of small decisions, not one big one.
Most systems fail at the seams rather than inside any one component. If two people in the business would answer this differently, that gap is the actual problem.
What it costs to ignore
Users notice inconsistency long before your monitoring does. In practice this is a scheduling problem more than a technical one. The practical test is whether someone new to the project could tell, in a minute, that it had been handled.
The steps
- Establish what you have today before changing anything
- Decide which reads must be immediate and which can lag
- Explain the delay in the interface rather than hiding it
- Write down the decision so the next person does not re-litigate it
Explain the delay in the interface rather than hiding it. Where this goes wrong is almost never a lack of knowledge. Doing this properly once is usually cheaper than doing it approximately three times.
A reasonable first step
What this looks like day to day
The right architecture for a team of three is the wrong one for a team of thirty, and vice versa. Three things worth confirming about eventual consistency before you move on:
- Someone can say what the current setup is without going to look
- Decide which reads must be immediate and which can lag — and you know whether that is true here
- There is a way to tell whether the last change to this helped
None of this needs a rewrite. Most of it is a morning's work once someone decides to do it.