When robots.txt is worth the effort

The version of this that works is simpler than the version most people imagine. Robots.txt is not free, and pretending otherwise leads to bad decisions.

Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. The teams that stay on top of it are the ones who put it on a calendar rather than a wish list.

When it is worth it

One careless line can deindex an entire site. None of that requires a large budget, only a decision and someone to own it. Most teams find the first pass takes an afternoon and the maintenance takes minutes a month.

When it is not

If nothing downstream depends on it and nobody is complaining, it can wait. Getting it slightly wrong is survivable. Ignoring it entirely is not.

How to decide

Point to your sitemap from it. It is worth being explicit about, because assumptions differ quietly. Doing this properly once is usually cheaper than doing it approximately three times.

In practice

Technical fixes remove obstacles; content earns the position. Both are needed and they are not interchangeable. Three things worth confirming about robots.txt before you move on:

• Someone can say what the current setup is without going to look
• One careless line can deindex an entire site — and you know whether that is true here
• There is a way to tell whether the last change to this helped

The point is not perfection, it is knowing which of these you have consciously chosen to skip.