Robots.txt Tester

Fetch any website's robots.txt file and check whether specific URL paths are allowed or blocked for crawlers.

The path to test, e.g. /blog/my-article or /admin

How robots.txt rules work.

Understanding rule precedence and path matching in the Robots Exclusion Protocol.

Everything

Disallow: /

Blocks the entire site.

/admin/ and all sub-paths

Disallow: /admin/

Blocks admin section only.

Everything

Allow: /

Allows the entire site (default behavior).

/admin/page only

Allow: /admin/page

Overrides a broader Disallow /admin/ for this specific path.

Nothing (empty value)

Disallow:

No restriction, equivalent to Allow: /.

All .pdf files

Disallow: /*.pdf$

Blocks all PDFs. Wildcards (*) and end-of-string ($) are supported.

Common robots.txt mistakes to avoid.

A misconfigured robots.txt can accidentally block your entire site from Google or leave sensitive directories exposed to unnecessary crawling. The most costly mistake is Disallow: / under User-agent: *, which blocks every crawler from crawling every page. This configuration is often left in place accidentally after development or staging environments are promoted to production.

Do not block CSS and JS

Googlebot needs access to your CSS and JavaScript files to render pages correctly. Blocking them prevents Google from seeing your content the way a browser would, which can hurt rankings.

Test before deploying

Use this tool to test your robots.txt rules before making them live. A single typo can accidentally block your most important pages from search engine crawlers.

Use Disallow sparingly

Blocking pages from crawling does not prevent them from being indexed if Google finds them through links. Only block pages you genuinely do not want crawled, such as admin areas, internal search results, or staging versions.

Add your sitemap URL

Including a Sitemap: directive in your robots.txt file tells crawlers where to find your sitemap automatically, without relying on manual submission in Google Search Console.

Frequently asked questions.

What is a robots.txt file?

A robots.txt file is a plain text file placed at the root of a website that tells search engine crawlers which pages or directories they are and are not allowed to access. It follows the Robots Exclusion Protocol, a voluntary standard recognized by Google, Bing, and all major crawlers. The file must be accessible at yourdomain.com/robots.txt.

Does robots.txt prevent pages from being indexed?

No. Robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in search results if Google discovers it through links from other sites. To prevent a page from being indexed, use a noindex meta tag in the page's HTML or an X-Robots-Tag: noindex HTTP header instead.

What is the difference between Allow and Disallow?

Disallow tells a crawler it cannot access a specific path. Allow overrides a Disallow rule to explicitly permit a path within a blocked section. For example, you can disallow /private/ but allow /private/public-page with an explicit Allow rule. When both rules apply, the more specific (longer) rule takes precedence.

How does robots.txt affect crawl budget?

Crawl budget is the number of pages Googlebot will crawl on your site in a given time window. Blocking low-value pages (admin panels, internal search results, duplicate content) in robots.txt prevents Googlebot from wasting crawl budget on them. This leaves more budget for your important pages.

What happens if I block CSS or JavaScript in robots.txt?

Blocking CSS and JS files from Googlebot prevents it from rendering your pages correctly, which hurts its ability to understand and index your content. Google explicitly recommends allowing crawlers to access all CSS, JavaScript, and image files that affect page rendering.

Is robots.txt a security tool?

No. Robots.txt is a voluntary protocol that well-behaved crawlers follow. Malicious bots, scrapers, and security scanners routinely ignore it. For sensitive content, use proper authentication and server-level access controls rather than relying on robots.txt.

Unlock Traffic on Autopilot

Grow organic traffic on autopilot. RankPill will research, write, optimize, and publish articles to your website daily.