How to get queues and message brokers right
Every audit we run turns up some version of this. The short answer to queues and message brokers is that it is mostly a sequence of small decisions, not one big one.
Architecture is the set of decisions that are expensive to reverse, which is the only reason they deserve the name. Assume whoever inherits this will have half your context and none of your patience.
The reason this keeps coming up
Queues absorb spikes that would otherwise take a system down. The reasoning matters more than the rule, because the rule has exceptions. It is the sort of thing that looks like polish right up until it costs you an enquiry.
The steps
- Establish what you have today before changing anything
- Every queue needs a dead letter destination and someone watching it
- Ordering guarantees are weaker than most teams assume
- Write down the decision so the next person does not re-litigate it
Ordering guarantees are weaker than most teams assume. This is the sort of thing that compounds, quietly, in both directions. Anything you cannot measure here, you are deciding by taste, which is fine as long as everyone knows it.
Turning this into a decision
In practice
The right architecture for a team of three is the wrong one for a team of thirty, and vice versa. Three things worth confirming about queues and message brokers before you move on:
- Someone can say what the current setup is without going to look
- Every queue needs a dead letter destination and someone watching it — and you know whether that is true here
- There is a way to tell whether the last change to this helped
The point is not perfection, it is knowing which of these you have consciously chosen to skip.