Before you invest in crawl budget

It comes up on almost every project, usually later than it should. Before you spend anything on crawl budget, it is worth confirming a few things are already true.

Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. The practical test is whether someone new to the project could tell, in a minute, that it had been handled.

Prerequisites

• You can describe the outcome you want in one sentence
• Someone owns it after the work is done
• Large sites can waste crawling on pages nobody should see
• You have a way to tell whether it worked

The mistakes we see most

Faceted navigation generates infinite URLs if unchecked. Getting it slightly wrong is survivable. Ignoring it entirely is not. Budget a little time for it every quarter and it never becomes a project of its own.

Clean internal linking is most of the fix. Getting it slightly wrong is survivable. Ignoring it entirely is not. Anything you cannot measure here, you are deciding by taste, which is fine as long as everyone knows it.

In practice

Technical fixes remove obstacles; content earns the position. Both are needed and they are not interchangeable. Three things worth confirming about crawl budget before you move on:

• Someone can say what the current setup is without going to look
• Faceted navigation generates infinite URLs if unchecked — and you know whether that is true here
• There is a way to tell whether the last change to this helped

Worth checking on your own setup before it becomes someone else's problem to fix.