How to get robots.txt right
The gap between knowing this and actually doing it is where most teams lose ground. The short answer to robots.txt is that it is mostly a sequence of small decisions, not one big one.
Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. The failure mode is not doing it wrong, it is doing it once and assuming it stays done.
Why it matters
One careless line can deindex an entire site. Where this goes wrong is almost never a lack of knowledge. Anything you cannot measure here, you are deciding by taste, which is fine as long as everyone knows it.
The steps
- Establish what you have today before changing anything
- Blocking a page does not remove it from results
- Point to your sitemap from it
- Write down the decision so the next person does not re-litigate it
Point to your sitemap from it. The reasoning matters more than the rule, because the rule has exceptions. Check it against what you would want a competitor's site to get wrong.
Turning this into a decision
The short version
Technical fixes remove obstacles; content earns the position. Both are needed and they are not interchangeable. Three things worth confirming about robots.txt before you move on:
- Someone can say what the current setup is without going to look
- Point to your sitemap from it — and you know whether that is true here
- There is a way to tell whether the last change to this helped
If you are not sure where your systems currently stand on this, it takes us about an hour to find out.