Crawl budget, explained without the jargon

The version of this that works is simpler than the version most people imagine. Here is crawl budget without the vocabulary that usually surrounds it.

Search engines are trying to answer a question, so the pages that answer questions clearly tend to do well. It is the sort of thing that looks like polish right up until it costs you an enquiry.

The short version

Large sites can waste crawling on pages nobody should see. Getting it slightly wrong is survivable. Ignoring it entirely is not. Check it against what you would want a competitor's site to get wrong.

Why people complicate it

Most of the confusion comes from tooling rather than from the idea itself. This is the sort of thing that compounds, quietly, in both directions.

Faceted navigation generates infinite URLs if unchecked. There is a version of this that is over-engineered, and it is worth avoiding. Doing this properly once is usually cheaper than doing it approximately three times.

Making it stick

Clean internal linking is most of the fix. None of that requires a large budget, only a decision and someone to own it. If it only works because one person remembers to do something, it does not work yet.

In practice

Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. Three things worth confirming about crawl budget before you move on:

• Someone can say what the current setup is without going to look
• Large sites can waste crawling on pages nobody should see — and you know whether that is true here
• There is a way to tell whether the last change to this helped

Worth checking on your own setup before it becomes someone else's problem to fix.