robots.txt is one of the smallest files on your site, and one of the easiest ways to cause an indexing incident. It controls crawler access, but it is not the same as "noindex". Understanding the difference between crawl controls and index controls is essential as your site grows.
robots.txt controls crawling, not indexing
If you disallow a page in robots.txt, crawlers may still index the URL if they discover it elsewhere (they just cannot fetch the content). If you need a page not to appear in search results, use an index control such as noindex (where supported) or remove the page entirely.
Keep robots.txt simple for public marketing sites
For most public sites, a minimal robots file is enough:
- Allow crawling for all agents.
- Reference your sitemap URL.
- Optionally disallow clearly non-public utility paths (admin, staging, temp directories).
Complex rules are harder to reason about and easier to break.
Use sitemaps to guide discovery
Sitemaps help crawlers find the pages you want discovered. Keep them clean and canonical-only (see sitemap hygiene).
Use canonical tags to consolidate duplicates
Canonical tags tell search engines which URL should receive credit when duplicates exist. They work best when you also keep internal links consistent (see canonical URLs on static sites).
Handle staging and preview environments explicitly
Many indexing accidents come from preview URLs. If you run staging environments:
- Prefer authentication over robots rules.
- If you must block, use both robots disallow and on-page noindex where feasible.
- Ensure preview hosts are not linked from the public site.
Build a quick "index control" checklist
A small checklist catches most issues:
- robots.txt references the right sitemap location.
- Sitemap contains the canonical URLs only.
- Each page has a single canonical and required meta tags.
- No accidental blocks for key paths (blog, services, contact).
Make it part of a regular technical SEO audit (see technical SEO audits).
If you keep robots.txt boring and consistent, you reduce the risk of the most painful SEO outage: your best pages disappearing from search.