RankPill Logo

Robots.txt Generator

Create a valid robots.txt file to control which pages Google and Bing can crawl and index on your site.

Crawl rules (User-agent: *)

Adds rules to block GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Bytespider, CCBot from scraping your content for AI model training.

User-agent: *
Allow: /

Common robots.txt examples.

Ready-to-use configurations for the most common website setups.

Allow everything (default)

Open to all crawlers. Standard for public websites.

User-agent: *
Allow: /

Block admin and private areas

Allow public crawling but protect backend paths.

User-agent: *
Disallow: /admin
Disallow: /wp-admin
Disallow: /private

Sitemap: https://example.com/sitemap.xml

Block AI training crawlers

Allow search engines but opt out of AI model training.

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Block all crawlers

Completely private site - no bot crawling of any kind.

User-agent: *
Disallow: /

What is a robots.txt file?

A robots.txt file is a plain text file placed at the root of your website (yourdomain.com/robots.txt) that instructs search engine crawlers which pages they can and cannot access. It follows the Robots Exclusion Protocol, a standard recognized by Google, Bing, and all major search engines.

While robots.txt is not a security tool and cannot prevent determined bots from accessing your site, it is an important part of technical SEO. Properly configured, it prevents crawl budget from being wasted on admin panels, duplicate content, and internal search result pages, keeping Google focused on the content you actually want indexed.

Protect admin areas

Disallow paths like /admin, /wp-admin, and /dashboard to keep backend pages out of search results and reduce unnecessary crawling.

Add your sitemap

Including your sitemap URL in robots.txt helps crawlers discover your sitemap automatically without relying on Google Search Console submission.

Do not block CSS or JS

Google needs to render your pages to understand them. Blocking stylesheets or JavaScript files can prevent proper indexing and hurt your rankings.

Test before deploying

Use Google Search Console's robots.txt tester to verify your file before going live. A misconfigured file can accidentally block your entire site from Google.

Frequently asked questions.

What is a robots.txt file?

A robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers which pages or sections they are allowed or not allowed to crawl. It follows the Robots Exclusion Protocol, a standard recognized by Google, Bing, and all major search engines.

Where do I upload my robots.txt file?

Upload robots.txt to the root of your website so it is accessible at yourdomain.com/robots.txt. In WordPress, it is usually at the root of your public_html or www folder. On Shopify or Webflow, it is managed through the platform settings.

Does robots.txt prevent pages from being indexed?

No. Robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in search results if other sites link to it. To prevent indexing, use a noindex meta tag or X-Robots-Tag header on the page itself.

Should I block AI crawlers from my website?

That depends on whether you want your content used to train AI models. Blocking GPTBot, ClaudeBot, and similar bots prevents them from scraping your content for training data. It does not affect your visibility in Google or Bing search results, which use separate crawlers.

What happens if I have no robots.txt file?

Crawlers will default to crawling your entire publicly accessible site. Most sites do not need a robots.txt file at all unless they have sections that should not be crawled, such as admin panels, duplicate content, or private pages.

What is crawl delay and should I use it?

Crawl delay tells bots to wait a set number of seconds between requests, which can reduce server load. Google ignores the crawl-delay directive. For Googlebot specifically, use Google Search Console to set crawl rate limits. Crawl delay is mainly useful for controlling Bingbot and smaller crawlers.

Unlock Traffic on Autopilot

Grow organic traffic on autopilot. RankPill will research, write, optimize, and publish articles to your website daily.