Robots.txt Generator (AI-aware)
Visual robots.txt generator with built-in presets for AI crawlers — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, PerplexityBot, Applebot-Extended — plus classic search engines, per-path controls, and sitemap field.
Global disallowed paths
Paths blocked for every crawler you enable. One per line. Example: /admin/, /private/, /cart.
Search engines
All crawlers (default) *
Wildcard rule applied to any bot without its own group
Googlebot Googlebot
Google search index
Bingbot Bingbot
Microsoft Bing search index
DuckDuckBot DuckDuckBot
DuckDuckGo search index
Baiduspider Baiduspider
Baidu search index
YandexBot YandexBot
Yandex search index
AI training crawlers
2023+These crawlers feed training data for ChatGPT, Claude, Gemini, and other LLMs. Blocking them opts your content out of model training.
GPTBot GPTBot
OpenAI training & retrieval (ChatGPT)
ClaudeBot ClaudeBot
Anthropic training & retrieval (Claude)
Google-Extended Google-Extended
Gemini training (separate from Googlebot)
CCBot CCBot
Common Crawl — open dataset used by many models
Bytespider Bytespider
ByteDance training crawler
PerplexityBot PerplexityBot
Perplexity answer engine retrieval
Applebot-Extended Applebot-Extended
Apple Intelligence training
ImagesiftBot ImagesiftBot
Image training (Glooma / Hive)
Sitemap
Generated robots.txt
User-agent: * Disallow:
How to use
- 1 For each crawler, choose Allow, Disallow, or custom paths.
- 2 Fill in your sitemap.xml URL and any global disallowed paths (e.g. /admin/, /private/).
- 3 Copy or download the generated robots.txt and upload it to your site root.
Related tools
About this tool
robots.txt is a tiny text file at your site root that tells crawlers — search engines, AI training bots, monitoring services — what they may and may not fetch. This generator produces a syntactically valid file without you having to remember the directives.
It ships with presets for every major search bot (Googlebot, Bingbot, DuckDuckBot, Baiduspider, YandexBot) and the new wave of AI training crawlers that emerged after 2023 (GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Gemini, CCBot for Common Crawl, Bytespider for ByteDance, PerplexityBot, Applebot-Extended, ImagesiftBot). You can allow or block any of them individually, and add global path rules that apply to everyone.
Download the result and drop it at https://yourdomain.com/robots.txt. Pair it with a sitemap.xml entry so search engines discover your full URL set faster.
Frequently asked questions
What happens if I block GPTBot, ClaudeBot, and the other AI crawlers? +
Blocking AI crawlers means your content will not be used to train the next generation of those companies' models. OpenAI (GPTBot), Anthropic (ClaudeBot), Google (Google-Extended for Gemini), and Common Crawl (CCBot, the data source behind many open-source models) all respect robots.txt. The trade-off is that your pages will not appear in answers from ChatGPT, Perplexity, or Gemini. That trade-off is a business decision only you can make — this tool does not pick a side.
Do all crawlers actually obey robots.txt? +
Major search engines (Google, Bing, Baidu, Yandex) and large AI labs honor robots.txt. Malicious scrapers, email harvesters, and small indexing services may ignore it. robots.txt is a convention, not a security mechanism — never use it to hide sensitive content. For real protection, use authentication, authorization, or noindex headers.
What happens when User-agent: * conflicts with a specific crawler block? +
Crawlers look for their own User-agent group first (Googlebot looks for 'User-agent: Googlebot'). Only if no specific group exists do they fall back to 'User-agent: *'. If you set a custom rule for a specific crawler, that crawler ignores the wildcard group entirely. This generator groups rules by crawler to prevent accidental conflicts.
What is the difference between 'Disallow: /' and 'Disallow:' (empty)? +
'Disallow: /' blocks access to everything under the root, which deindexes your whole site. 'Disallow:' with nothing after the colon blocks nothing, which is equivalent to allowing all paths. The two are easy to confuse and the mistake can drop your site from search results. This generator sidesteps the issue with explicit Allow / Disallow / Custom buttons.
Is the sitemap field required? +
No, but it is strongly recommended. Declaring 'Sitemap: https://.../sitemap.xml' in robots.txt helps search engines discover all your pages faster — especially valuable for new domains, deeply nested content, or sites with weak internal linking. Multiple sitemaps can be declared on separate Sitemap: lines.