Robots.txt Tester
Fetch any website's robots.txt file and check whether specific URL paths are allowed or blocked for crawlers.
The path to test, e.g. /blog/my-article or /admin
How robots.txt Rules Work
Understanding rule precedence and path matching in the Robots Exclusion Protocol.
| Rule | Applies to path | Result |
|---|---|---|
| Disallow: / | Everything | Blocks the entire site. |
| Disallow: /admin/ | /admin/ and all sub-paths | Blocks admin section only. |
| Allow: / | Everything | Allows the entire site (default behavior). |
| Allow: /admin/page | /admin/page only | Overrides a broader Disallow /admin/ for this specific path. |
| Disallow: | Nothing (empty value) | No restriction — equivalent to Allow: /. |
| Disallow: /*.pdf$ | All .pdf files | Blocks all PDFs. Wildcards (*) and end-of-string ($) are supported. |
Common robots.txt Mistakes to Avoid
A misconfigured robots.txt can accidentally block your entire site from Google or leave sensitive directories exposed to unnecessary crawling. The most costly mistake is Disallow: / under User-agent: *, which blocks every crawler from crawling every page. This configuration is often left in place accidentally after development or staging environments are promoted to production.
Do not block CSS and JS
Googlebot needs access to your CSS and JavaScript files to render pages correctly. Blocking them prevents Google from seeing your content the way a browser would, which can hurt rankings.
Test before deploying
Use this tool to test your robots.txt rules before making them live. A single typo can accidentally block your most important pages from search engine crawlers.
Use Disallow sparingly
Blocking pages from crawling does not prevent them from being indexed if Google finds them through links. Only block pages you genuinely do not want crawled, such as admin areas, internal search results, or staging versions.
Add your sitemap URL
Including a Sitemap: directive in your robots.txt file tells crawlers where to find your sitemap automatically, without relying on manual submission in Google Search Console.
Frequently Asked Questions
Common questions about robots.txt and crawler directives.
Related Tools
More free analysis tools
XML Sitemap Validator
Validate your XML sitemap for syntax errors, broken URLs, and missing pages that could prevent proper Google indexing.
Canonical Tag Checker
Check any URL for canonical tag issues that cause duplicate content problems and dilute your page's search ranking signals.
Keyword Density Checker
Measure keyword frequency in any content and identify over-optimization or under-usage before publishing to search engines.
Open Graph Checker
Inspect any URL's Open Graph tags, meta description, and Twitter Card data to verify correct social previews and SEO output.
Want these on autopilot?
RankPill automates everything these tools do. Meta descriptions, titles, content briefs, and full articles published to your site every day without lifting a finger.
