Robots.txt Tester
Fetch any website's robots.txt file and check whether specific URL paths are allowed or blocked for crawlers.
The path to test, e.g. /blog/my-article or /admin
How robots.txt Rules Work
Understanding rule precedence and path matching in the Robots Exclusion Protocol.
| Rule | Applies to path | Result |
|---|---|---|
| Disallow: / | Everything | Blocks the entire site. |
| Disallow: /admin/ | /admin/ and all sub-paths | Blocks admin section only. |
| Allow: / | Everything | Allows the entire site (default behavior). |
| Allow: /admin/page | /admin/page only | Overrides a broader Disallow /admin/ for this specific path. |
| Disallow: | Nothing (empty value) | No restriction, equivalent to Allow: /. |
| Disallow: /*.pdf$ | All .pdf files | Blocks all PDFs. Wildcards (*) and end-of-string ($) are supported. |
Common robots.txt Mistakes to Avoid
A misconfigured robots.txt can accidentally block your entire site from Google or leave sensitive directories exposed to unnecessary crawling. The most costly mistake is Disallow: / under User-agent: *, which blocks every crawler from crawling every page. This configuration is often left in place accidentally after development or staging environments are promoted to production.
Do not block CSS and JS
Googlebot needs access to your CSS and JavaScript files to render pages correctly. Blocking them prevents Google from seeing your content the way a browser would, which can hurt rankings.
Test before deploying
Use this tool to test your robots.txt rules before making them live. A single typo can accidentally block your most important pages from search engine crawlers.
Use Disallow sparingly
Blocking pages from crawling does not prevent them from being indexed if Google finds them through links. Only block pages you genuinely do not want crawled, such as admin areas, internal search results, or staging versions.
Add your sitemap URL
Including a Sitemap: directive in your robots.txt file tells crawlers where to find your sitemap automatically, without relying on manual submission in Google Search Console.
Frequently Asked Questions
Common questions about robots.txt and crawler directives.
