The real cost of ignoring robots.txt
It is rarely the thing that gets a project approved, and often the thing that decides how it goes. Nobody bills you for neglecting robots.txt. The cost shows up somewhere else.
Technical fixes remove obstacles; content earns the position. Both are needed and they are not interchangeable. If two people in the business would answer this differently, that gap is the actual problem.
Where the cost lands
- Time spent on work that should not have been necessary
- Enquiries that quietly never arrive
- One careless line can deindex an entire site
- Rework, once the problem is finally visible
Blocking a page does not remove it from results. This is the sort of thing that compounds, quietly, in both directions. Write the reasoning down alongside the decision, because the reasoning is what changes first.
A reasonable first step
Point to your sitemap from it. The reasoning matters more than the rule, because the rule has exceptions. If it only works because one person remembers to do something, it does not work yet.
What this looks like day to day
Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. Three things worth confirming about robots.txt before you move on:
- Someone can say what the current setup is without going to look
- Point to your sitemap from it — and you know whether that is true here
- There is a way to tell whether the last change to this helped
If you are not sure where your systems currently stand on this, it takes us about an hour to find out.