Skip to content

Free tool

robots.txt generator for AI crawlers.

Choose what each kind of AI crawler may do. The file updates as you choose.

Start from

Model training

Collects pages that may be used to train models.

  • GPTBotOpenAI
  • ClaudeBotAnthropic
  • Google-ExtendedGoogle

Blocking Google-Extended also limits grounding in Gemini apps, not only training. It has no effect on Google Search.

AI search

Builds the index that AI answers search and cite.

  • OAI-SearchBotOpenAI
  • Claude-SearchBotAnthropic
  • PerplexityBotPerplexity
User-triggered

Fetches a page when a person asks the assistant about it.

  • ChatGPT-UserOpenAI
  • Claude-UserAnthropic
  • Perplexity-UserPerplexity

OpenAI and Perplexity say robots.txt may not apply to their user-triggered fetchers.

Your robots.txt

# Allowed
User-agent: Google-Extended
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /

# Blocked
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /

robots.txt is a request, not a lock. Well-behaved crawlers follow it; it does not protect private content.

How to use it

  1. 1Save the text as robots.txt.
  2. 2Put it at the root of your site, so it opens at yourbusiness.com/robots.txt.
  3. 3If you already have a robots.txt, merge the groups instead of replacing the file.
After you publish it, check the resultCan AI read your site?