A short guide to robots.txt

The gap between knowing this and actually doing it is where most teams lose ground. Everything we would tell a client about robots.txt in the time it takes to drink a coffee.

Technical fixes remove obstacles; content earns the position. Both are needed and they are not interchangeable. The failure mode is not doing it wrong, it is doing it once and assuming it stays done.

The reason this keeps coming up

One careless line can deindex an entire site. This is the sort of thing that compounds, quietly, in both directions. Budget a little time for it every quarter and it never becomes a project of its own.

What good looks like

Blocking a page does not remove it from results. That sounds obvious written down. It is still the thing most often skipped. Write the reasoning down alongside the decision, because the reasoning is what changes first.

Common failure modes

Point to your sitemap from it. In practice this is a scheduling problem more than a technical one. The failure mode is not doing it wrong, it is doing it once and assuming it stays done.

How to tell if yours is fine

Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. Three things worth confirming about robots.txt before you move on:

• Someone can say what the current setup is without going to look
• One careless line can deindex an entire site — and you know whether that is true here
• There is a way to tell whether the last change to this helped

Pick the one that would hurt most if it failed, and start there.