Getting started with robots.txt
This is cheap to get right at the start and expensive to retrofit. A short on-ramp to robots.txt for teams who have not touched it before.
Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. Anything you cannot measure here, you are deciding by taste, which is fine as long as everyone knows it.
What it costs to ignore
One careless line can deindex an entire site. Where this goes wrong is almost never a lack of knowledge. The practical test is whether someone new to the project could tell, in a minute, that it had been handled.
Your first week
- Find out what is already in place
- Blocking a page does not remove it from results
- Change one thing and measure it
Point to your sitemap from it. This is the sort of thing that compounds, quietly, in both directions. Assume whoever inherits this will have half your context and none of your patience.
In practice
Technical fixes remove obstacles; content earns the position. Both are needed and they are not interchangeable. Three things worth confirming about robots.txt before you move on:
- Someone can say what the current setup is without going to look
- Blocking a page does not remove it from results — and you know whether that is true here
- There is a way to tell whether the last change to this helped
If you want a second opinion on how yours is set up, ask.