robots.txt Generator

How robots.txt actually works

robots.txt is a plain-text file served from your site's root (it must be exactly https://example.com/robots.txt — a subdirectory won't work). Crawlers fetch it before crawling and match themselves against User-agent groups: a bot uses the most specific group that names it, and falls back to the * group otherwise — the groups don't combine. Within a group, Disallow lines list path prefixes the bot shouldn't fetch and Allow lines carve out exceptions (Google resolves conflicts by longest matching rule). An empty Disallow: means "nothing is disallowed" — that's the canonical way to explicitly allow everything. Crucially, robots.txt is advisory: well-behaved crawlers honor it, but it is not access control. Anything truly private needs authentication, not a Disallow line — in fact, listing secret paths in robots.txt advertises them.

Blocking AI training crawlers

A growing set of bots crawl the web to gather LLM training data, and the major ones respect robots.txt: GPTBot (OpenAI), CCBot (Common Crawl, whose corpus feeds many models), Google-Extended (a token that controls Gemini training without affecting Google Search indexing), anthropic-ai and PerplexityBot. The "Block AI crawlers" preset adds a Disallow: / group for each of them while leaving the * group open, so regular search engines keep indexing your site normally. Because groups are matched per-agent and don't inherit, each bot needs its own group — which is exactly what the preset generates.

FAQ

Does Disallow remove my pages from Google?

Not reliably. Disallow stops crawling, but a URL that's linked from elsewhere can still appear in results as a bare link with no snippet. To keep a page out of the index, let it be crawled and give it a noindex meta tag (or X-Robots-Tag header) instead — and don't block it in robots.txt, or Google will never see the noindex.

What does Crawl-delay do, and who honors it?

Crawl-delay: 10 asks a bot to wait 10 seconds between requests. Bing, Yandex and several others honor it; Googlebot ignores it entirely (Google manages crawl rate automatically via Search Console). It's a polite hint for reducing load from aggressive crawlers, not a guarantee.

Should I include a Sitemap line?

Yes — Sitemap: https://example.com/sitemap.xml is the easiest way to point every crawler at your sitemap without registering with each search engine's webmaster tools. It's independent of the User-agent groups and can appear anywhere in the file; this tool puts it at the end by convention.