Getting started with crawl budget
Teams tend to reach for this after something has already gone wrong. A short on-ramp to crawl budget for teams who have not touched it before.
Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. Doing this properly once is usually cheaper than doing it approximately three times.
What is actually at stake
Large sites can waste crawling on pages nobody should see. There is a version of this that is over-engineered, and it is worth avoiding. Most teams find the first pass takes an afternoon and the maintenance takes minutes a month.
Your first week
- Find out what is already in place
- Faceted navigation generates infinite URLs if unchecked
- Change one thing and measure it
Clean internal linking is most of the fix. The reasoning matters more than the rule, because the rule has exceptions. If two people in the business would answer this differently, that gap is the actual problem.
How to tell if yours is fine
Search engines are trying to answer a question, so the pages that answer questions clearly tend to do well. Three things worth confirming about crawl budget before you move on:
- Someone can say what the current setup is without going to look
- Large sites can waste crawling on pages nobody should see — and you know whether that is true here
- There is a way to tell whether the last change to this helped
Most of the value here comes from doing the first two things, not all of them.