Crawl budget: the questions we get asked most

Every audit we run turns up some version of this. The questions about crawl budget that come up most often on our calls.

Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. It is worth deciding this deliberately rather than inheriting whatever the last person set up.

Do we need to care about this?

Large sites can waste crawling on pages nobody should see. Small and consistent beats large and occasional here. Doing this properly once is usually cheaper than doing it approximately three times.

Can it wait until after launch?

Occasionally. More often the post-launch version costs several times the pre-launch one. This is the sort of thing that compounds, quietly, in both directions.

How do we know it is working?

Clean internal linking is most of the fix. The cost of getting this wrong is rarely visible on the day it happens. Assume whoever inherits this will have half your context and none of your patience.

How to tell if yours is fine

Technical fixes remove obstacles; content earns the position. Both are needed and they are not interchangeable. Three things worth confirming about crawl budget before you move on:

• Someone can say what the current setup is without going to look
• Large sites can waste crawling on pages nobody should see — and you know whether that is true here
• There is a way to tell whether the last change to this helped

If any of that sounds like a description of your current setup, it is fixable.