Signs it is time to revisit robots.txt

Teams tend to reach for this after something has already gone wrong. A few signals that robots.txt is due some attention.

Technical fixes remove obstacles; content earns the position. Both are needed and they are not interchangeable. It is the sort of thing that looks like polish right up until it costs you an enquiry.

The signals

• Nobody can say when it was last reviewed
• The answer depends on who you ask
• One careless line can deindex an entire site
• Blocking a page does not remove it from results

The practical version

Point to your sitemap from it. None of that requires a large budget, only a decision and someone to own it. It is worth deciding this deliberately rather than inheriting whatever the last person set up.

In practice

Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. Three things worth confirming about robots.txt before you move on:

• Someone can say what the current setup is without going to look
• Blocking a page does not remove it from results — and you know whether that is true here
• There is a way to tell whether the last change to this helped

If any of that sounds like a description of your current setup, it is fixable.