Glossary /Technical SEO
Crawl Budget
The number of URLs a search engine will fetch from a site in a given period, set by its capacity and its appetite.
Crawl budget is the practical ceiling on how much of a site gets fetched. It is a function of two things: how much load the server can take without degrading, and how much the engine thinks the site is worth re-checking. Neither is directly controllable, but both are influenced.
For most sites this is a non-issue and worrying about it is a distraction. It becomes real above roughly the ten-thousand-URL mark, or wherever a site generates unbounded URLs — faceted navigation, calendars, search result pages, parameter combinations — because the crawler spends its allowance on permutations instead of pages that matter.
When it does bite, the fix is subtraction. Stop generating the junk URLs, remove redirect chains, return proper 404s instead of soft ones, and keep the sitemap honest so the crawler has a reliable list of what is worth its time.