Five mistakes teams make with crawl budget

The version of this that works is simpler than the version most people imagine. These are the ones we run into repeatedly when we audit crawl budget.

Search engines are trying to answer a question, so the pages that answer questions clearly tend to do well. The version that survives contact with a real deadline is the simple one.

Where it usually goes wrong

• Treating it as a launch task rather than an ongoing one
• Assuming someone else already owns it
• Large sites can waste crawling on pages nobody should see
• Faceted navigation generates infinite URLs if unchecked
• Never checking whether the fix actually worked

Clean internal linking is most of the fix. Small and consistent beats large and occasional here. The teams that stay on top of it are the ones who put it on a calendar rather than a wish list.

Turning this into a decision

How to tell if yours is fine

Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. Three things worth confirming about crawl budget before you move on:

• Someone can say what the current setup is without going to look
• Large sites can waste crawling on pages nobody should see — and you know whether that is true here
• There is a way to tell whether the last change to this helped

The point is not perfection, it is knowing which of these you have consciously chosen to skip.