Background jobs: what to get right first

Every audit we run turns up some version of this. If you only fix one thing about background jobs this quarter, make it the first item below.

Code gets read far more often than it gets written, and usually by someone with less context than the author had. It rarely shows up as a line item, which is exactly why it slips.

Start here

Anything slow should happen outside the request cycle. There is a version of this that is over-engineered, and it is worth avoiding. Doing this properly once is usually cheaper than doing it approximately three times.

Then this

Jobs need retries, because networks fail. Small and consistent beats large and occasional here. Write the reasoning down alongside the decision, because the reasoning is what changes first.

Eventually

Monitor the queue depth or failures pile up unnoticed. There is a version of this that is over-engineered, and it is worth avoiding. Doing this properly once is usually cheaper than doing it approximately three times.

In practice

The question is rarely whether something can be built, but what it costs to keep running afterwards. Three things worth confirming about background jobs before you move on:

• Someone can say what the current setup is without going to look
• Monitor the queue depth or failures pile up unnoticed — and you know whether that is true here
• There is a way to tell whether the last change to this helped

Pick the one that would hurt most if it failed, and start there.