Robots.txt: the questions we get asked most

We end up explaining this on discovery calls often enough that it deserved writing down. The questions about robots.txt that come up most often on our calls.

Technical fixes remove obstacles; content earns the position. Both are needed and they are not interchangeable. Budget a little time for it every quarter and it never becomes a project of its own.

Do we need to care about this?

One careless line can deindex an entire site. The reasoning matters more than the rule, because the rule has exceptions. It is worth deciding this deliberately rather than inheriting whatever the last person set up.

Can it wait until after launch?

Occasionally. More often the post-launch version costs several times the pre-launch one. That sounds obvious written down. It is still the thing most often skipped.

How do we know it is working?

Point to your sitemap from it. The reasoning matters more than the rule, because the rule has exceptions. Doing this properly once is usually cheaper than doing it approximately three times.

How to tell if yours is fine

Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. Three things worth confirming about robots.txt before you move on:

• Someone can say what the current setup is without going to look
• Blocking a page does not remove it from results — and you know whether that is true here
• There is a way to tell whether the last change to this helped

If any of that sounds like a description of your current setup, it is fixable.