Glossary /Foundations
Crawling
The process by which a search engine discovers URLs and fetches them, which has to happen before a page can rank for anything.
Crawling is discovery and fetching. A bot follows links, reads sitemaps, and requests URLs it has not seen or has not checked recently. Nothing else in SEO matters if this step fails: a page that is never fetched cannot be indexed, and a page that is not indexed cannot rank.
Most crawl problems are self-inflicted and boring. A blocked path in robots.txt, a chain of redirects that the bot abandons, a link built in JavaScript that never renders as an anchor, a server that times out under bot load. None of them announce themselves in a rankings report — they show up as pages that simply never appear.
Crawling is not indexing, and conflating the two causes a specific mistake: blocking a URL in robots.txt to get it out of the index. The block stops the fetch, which means the engine never reads the noindex tag you put there, and the URL can persist in results with no description under it.