Robots.txt Configuration Guide for Crawl Budget Optimization
Prevent search engines from wasting crawl budget on dynamic cart pages, search endpoints, and private scripts.
Controlling Crawler Access
Use precise Disallow and Allow directives in your robots.txt file to keep bots focused strictly on valuable indexable pages.