Free tool
robots.txt generator for AI crawlers.
Choose what each kind of AI crawler may do. The file updates as you choose.
Start from
Your robots.txt
# Allowed
User-agent: Google-Extended
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /
# Blocked
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /
# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /
robots.txt is a request, not a lock. Well-behaved crawlers follow it; it does not protect private content.
How to use it
- 1Save the text as robots.txt.
- 2Put it at the root of your site, so it opens at yourbusiness.com/robots.txt.
- 3If you already have a robots.txt, merge the groups instead of replacing the file.