Robots.txt: what to get right first

This is cheap to get right at the start and expensive to retrofit. If you only fix one thing about robots.txt this quarter, make it the first item below.

Technical fixes remove obstacles; content earns the position. Both are needed and they are not interchangeable. Anything you cannot measure here, you are deciding by taste, which is fine as long as everyone knows it.

Start here

One careless line can deindex an entire site. That sounds obvious written down. It is still the thing most often skipped. If two people in the business would answer this differently, that gap is the actual problem.

Then this

Blocking a page does not remove it from results. Getting it slightly wrong is survivable. Ignoring it entirely is not. Anything you cannot measure here, you are deciding by taste, which is fine as long as everyone knows it.

Eventually

Point to your sitemap from it. The cost of getting this wrong is rarely visible on the day it happens. Doing this properly once is usually cheaper than doing it approximately three times.

In practice

Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. Three things worth confirming about robots.txt before you move on:

• Someone can say what the current setup is without going to look
• One careless line can deindex an entire site — and you know whether that is true here
• There is a way to tell whether the last change to this helped

If you want a second opinion on how yours is set up, ask.