How to get crawl budget right
The version of this that works is simpler than the version most people imagine. The short answer to crawl budget is that it is mostly a sequence of small decisions, not one big one.
Search engines are trying to answer a question, so the pages that answer questions clearly tend to do well. Anything you cannot measure here, you are deciding by taste, which is fine as long as everyone knows it.
Why this earns attention
Large sites can waste crawling on pages nobody should see. The cost of getting this wrong is rarely visible on the day it happens. Most teams find the first pass takes an afternoon and the maintenance takes minutes a month.
The steps
- Establish what you have today before changing anything
- Faceted navigation generates infinite URLs if unchecked
- Clean internal linking is most of the fix
- Write down the decision so the next person does not re-litigate it
Clean internal linking is most of the fix. None of that requires a large budget, only a decision and someone to own it. Assume whoever inherits this will have half your context and none of your patience.
What to do next
The short version
Technical fixes remove obstacles; content earns the position. Both are needed and they are not interchangeable. Three things worth confirming about crawl budget before you move on:
- Someone can say what the current setup is without going to look
- Clean internal linking is most of the fix — and you know whether that is true here
- There is a way to tell whether the last change to this helped
The point is not perfection, it is knowing which of these you have consciously chosen to skip.