Five mistakes teams make with robots.txt

This is cheap to get right at the start and expensive to retrofit. These are the ones we run into repeatedly when we audit robots.txt.

Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. Assume whoever inherits this will have half your context and none of your patience.

Common failure modes

• Treating it as a launch task rather than an ongoing one
• Assuming someone else already owns it
• One careless line can deindex an entire site
• Blocking a page does not remove it from results
• Never checking whether the fix actually worked

Point to your sitemap from it. This is the sort of thing that compounds, quietly, in both directions. Assume whoever inherits this will have half your context and none of your patience.

What to do next

In practice

Technical fixes remove obstacles; content earns the position. Both are needed and they are not interchangeable. Three things worth confirming about robots.txt before you move on:

• Someone can say what the current setup is without going to look
• One careless line can deindex an entire site — and you know whether that is true here
• There is a way to tell whether the last change to this helped

If you want a second opinion on how yours is set up, ask.