Before you invest in robots.txt

Teams tend to reach for this after something has already gone wrong. Before you spend anything on robots.txt, it is worth confirming a few things are already true.

Search engines are trying to answer a question, so the pages that answer questions clearly tend to do well. The version that survives contact with a real deadline is the simple one.

Prerequisites

• You can describe the outcome you want in one sentence
• Someone owns it after the work is done
• One careless line can deindex an entire site
• You have a way to tell whether it worked

Common failure modes

Blocking a page does not remove it from results. Where this goes wrong is almost never a lack of knowledge. It is the sort of thing that looks like polish right up until it costs you an enquiry.

Point to your sitemap from it. The teams that handle this well are rarely the ones with the biggest budgets. Doing this properly once is usually cheaper than doing it approximately three times.

The short version

Technical fixes remove obstacles; content earns the position. Both are needed and they are not interchangeable. Three things worth confirming about robots.txt before you move on:

• Someone can say what the current setup is without going to look
• Blocking a page does not remove it from results — and you know whether that is true here
• There is a way to tell whether the last change to this helped

If you want a second opinion on how yours is set up, ask.