SEO
Crawl Budget
Crawl budget is the number of pages a search engine bot will crawl on your site within a given timeframe. Managing crawl budget is important for large websites to ensure important pages are indexed.
Examples
An ecommerce site with 500,000 product pages optimizes its crawl budget by blocking search engines from indexing filtered and faceted navigation pages.
Best Practices
Use robots.txt, sitemaps, and internal linking strategically to direct crawl budget toward your most valuable pages.