What is a robots.txt file?
A robots.txt file is a plain text file placed at the root of a website that tells search engine crawlers which pages or directories they are and are not allowed to access. It follows the Robots Exclusion Protocol, a voluntary standard recognized by Google, Bing, and all major crawlers. The file must be accessible at yourdomain.com/robots.txt.
Does robots.txt prevent pages from being indexed?
No. Robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in search results if Google discovers it through links from other sites. To prevent a page from being indexed, use a noindex meta tag in the page's HTML or an X-Robots-Tag: noindex HTTP header instead.
What is the difference between Allow and Disallow?
Disallow tells a crawler it cannot access a specific path. Allow overrides a Disallow rule to explicitly permit a path within a blocked section. For example, you can disallow /private/ but allow /private/public-page with an explicit Allow rule. When both rules apply, the more specific (longer) rule takes precedence.
How does robots.txt affect crawl budget?
Crawl budget is the number of pages Googlebot will crawl on your site in a given time window. Blocking low-value pages (admin panels, internal search results, duplicate content) in robots.txt prevents Googlebot from wasting crawl budget on them. This leaves more budget for your important pages.
What happens if I block CSS or JavaScript in robots.txt?
Blocking CSS and JS files from Googlebot prevents it from rendering your pages correctly, which hurts its ability to understand and index your content. Google explicitly recommends allowing crawlers to access all CSS, JavaScript, and image files that affect page rendering.
Is robots.txt a security tool?
No. Robots.txt is a voluntary protocol that well-behaved crawlers follow. Malicious bots, scrapers, and security scanners routinely ignore it. For sensitive content, use proper authentication and server-level access controls rather than relying on robots.txt.