Robots.txt for small teams

This is one of those topics that looks small until it costs you something. Most advice about robots.txt assumes a team that does not exist at your size. Here is the version that does not.

Search engines are trying to answer a question, so the pages that answer questions clearly tend to do well. Anything you cannot measure here, you are deciding by taste, which is fine as long as everyone knows it.

What to keep

One careless line can deindex an entire site. The teams that handle this well are rarely the ones with the biggest budgets. It rarely shows up as a line item, which is exactly why it slips.

What to drop

Process that exists to coordinate ten people is overhead when there are two of you. Where this goes wrong is almost never a lack of knowledge.

Where to start

Point to your sitemap from it. Small and consistent beats large and occasional here. If it only works because one person remembers to do something, it does not work yet.

The short version

Search work compounds slowly, which is why it gets abandoned about two months before it would have paid off. Three things worth confirming about robots.txt before you move on:

• Someone can say what the current setup is without going to look
• Blocking a page does not remove it from results — and you know whether that is true here
• There is a way to tell whether the last change to this helped

If you want a second opinion on how yours is set up, ask.