A short guide to crawl budget

It is rarely the thing that gets a project approved, and often the thing that decides how it goes. Everything we would tell a client about crawl budget in the time it takes to drink a coffee.

Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. The failure mode is not doing it wrong, it is doing it once and assuming it stays done.

Why this earns attention

Large sites can waste crawling on pages nobody should see. Where this goes wrong is almost never a lack of knowledge. The failure mode is not doing it wrong, it is doing it once and assuming it stays done.

How to approach it

Faceted navigation generates infinite URLs if unchecked. It is worth being explicit about, because assumptions differ quietly. Assume whoever inherits this will have half your context and none of your patience.

Warning signs

Clean internal linking is most of the fix. This is the sort of thing that compounds, quietly, in both directions. Check it against what you would want a competitor's site to get wrong.

How to tell if yours is fine

Search engines are trying to answer a question, so the pages that answer questions clearly tend to do well. Three things worth confirming about crawl budget before you move on:

• Someone can say what the current setup is without going to look
• Faceted navigation generates infinite URLs if unchecked — and you know whether that is true here
• There is a way to tell whether the last change to this helped

If you want a second opinion on how yours is set up, ask.