Three myths about robots.txt
It is rarely the thing that gets a project approved, and often the thing that decides how it goes. A few things about robots.txt that get repeated more often than they get checked.
Technical fixes remove obstacles; content earns the position. Both are needed and they are not interchangeable. If it only works because one person remembers to do something, it does not work yet.
“It only matters for big sites”
One careless line can deindex an entire site. The reasoning matters more than the rule, because the rule has exceptions. The teams that stay on top of it are the ones who put it on a calendar rather than a wish list.
“We can deal with it after launch”
Sometimes true, usually expensive. It is worth being explicit about, because assumptions differ quietly.
“Our platform handles it”
Point to your sitemap from it. It is worth being explicit about, because assumptions differ quietly. The version that survives contact with a real deadline is the simple one.
How to tell if yours is fine
Search engines are trying to answer a question, so the pages that answer questions clearly tend to do well. Three things worth confirming about robots.txt before you move on:
- Someone can say what the current setup is without going to look
- Point to your sitemap from it — and you know whether that is true here
- There is a way to tell whether the last change to this helped
Most of the value here comes from doing the first two things, not all of them.