Glossary /Technical SEO
robots.txt
A file at the root of a domain that tells compliant crawlers which paths they may request. It controls fetching, not indexing.
robots.txt lives at the root of a domain and lists paths that crawlers should not request. It is a request, not an enforcement mechanism — well-behaved bots honour it and the rest ignore it, so it is a crawl management tool rather than a security control. Anything genuinely private needs authentication.
The distinction that catches people out: robots.txt governs fetching, not indexing. A disallowed URL can still appear in results if other pages link to it, because the engine learned the URL exists without ever reading it. To keep a page out of the index you must let it be fetched and serve a noindex directive, which is the opposite of the intuitive move.
It is also a file where small syntax errors are expensive and silent. A stray directive, a wildcard that matches more than intended, or a non-standard line that makes a validator reject the whole file can remove sections of a site from search without any other symptom.