Robots.txt, explained without the jargon

The gap between knowing this and actually doing it is where most teams lose ground. Here is robots.txt without the vocabulary that usually surrounds it.

Search engines are trying to answer a question, so the pages that answer questions clearly tend to do well. The practical test is whether someone new to the project could tell, in a minute, that it had been handled.

The short version

One careless line can deindex an entire site. Where this goes wrong is almost never a lack of knowledge. It rarely shows up as a line item, which is exactly why it slips.

Why people complicate it

Most of the confusion comes from tooling rather than from the idea itself. Getting it slightly wrong is survivable. Ignoring it entirely is not.

Blocking a page does not remove it from results. Getting it slightly wrong is survivable. Ignoring it entirely is not. Check it against what you would want a competitor's site to get wrong.

Where to go from here

Point to your sitemap from it. The cost of getting this wrong is rarely visible on the day it happens. Assume whoever inherits this will have half your context and none of your patience.

The short version

Technical fixes remove obstacles; content earns the position. Both are needed and they are not interchangeable. Three things worth confirming about robots.txt before you move on:

• Someone can say what the current setup is without going to look
• Blocking a page does not remove it from results — and you know whether that is true here
• There is a way to tell whether the last change to this helped

The point is not perfection, it is knowing which of these you have consciously chosen to skip.