Skip to content
All articles

robots.txt, llms.txt and AI crawlers: what a small site should set up

By Rastock AEO teamPublished 6 min read

A small site should decide on purpose which AI crawlers it allows in robots.txt, using each company's official user-agent names, and make sure its key facts are in the HTML the server sends. An llms.txt file is optional: it is a community proposal, and Google Search says it does not use it.

AI companies read websites with automated programs called crawlers. Each crawler announces a name, called a user agent, and most check a text file on your site called robots.txt before reading anything. This guide covers the names that matter, a safe starting setup, and two topics that cause confusion: llms.txt and JavaScript.

Three kinds of AI crawlers

The big providers split their crawlers by purpose. The split matters because blocking one kind has a different effect from blocking another.

  • Training crawlers collect pages that may be used to train future models. Blocking them is an opt-out from training. It does not remove you from AI search.
  • Search crawlers build the index an assistant searches when it answers with web results. Blocking them can keep your pages out of those answers and their source links.
  • User-triggered fetchers visit a page because a person asked the assistant about it or pasted its address. They are not automatic crawling, and some providers say robots.txt rules may not apply to them.

The official user-agent names

These names come from each company's own documentation, checked in November 2026. They change from time to time, so follow the source links before you edit a live file.

CompanyName in robots.txtPurposeWhat the company says
OpenAIGPTBotTrainingDisallowing it signals your content should not be used to train its models.
OpenAIOAI-SearchBotSearchSites that opt out are not shown in ChatGPT search answers, though they may still appear as plain navigational links.
OpenAIChatGPT-UserUser-triggeredNot used for automatic crawling. Robots.txt rules may not apply.
AnthropicClaudeBotTrainingBlocking it excludes future material from training datasets.
AnthropicClaude-SearchBotSearchBlocking it may reduce visibility and accuracy in Claude's search results.
AnthropicClaude-UserUser-triggeredHonors robots.txt. Blocking it stops Claude fetching your page for a user's question.
PerplexityPerplexityBotSearchUsed to surface and link sites in Perplexity. Not used for model training.
PerplexityPerplexity-UserUser-triggeredGenerally ignores robots.txt, because a person asked for the fetch.
GoogleGooglebotSearchFeeds Google Search. Google says its AI features draw on the Search index.
GoogleGoogle-ExtendedControl name onlyNot a separate crawler. Controls Gemini training and grounding in Gemini apps. No effect on Google Search.

Two notes. OpenAI also lists OAI-AdsBot, which only visits pages submitted as ads on ChatGPT. And Microsoft's posts about Copilot do not name a separate AI crawler. They address its crawler as Bingbot and use page-level tags, covered below.

A robots.txt example

The file lives at the root of your domain, for example https://example.com/robots.txt. A common choice for a business that wants to be found is: allow search and user-triggered access, opt out of training. That looks like this:

# AI search and user-triggered fetches: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /

# Model training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

To allow training as well, delete the second group. To block a crawler from one folder only, replace / with a path such as /members/.

Think before adding Google-Extended to the blocked group. Google says that name covers both Gemini training and grounding in Gemini apps, so blocking it may also limit how the Gemini app uses your pages when answering.

What robots.txt cannot do

  • It is a request, not a lock. Google's guide says the file cannot enforce behavior, and that it is up to each crawler to obey. Put private content behind a login.
  • Changes are not instant. OpenAI and Perplexity both say their systems can take about 24 hours to reflect an update.
  • Other layers can block silently. A firewall, CDN or host may refuse AI crawlers before they ever read your file. Ask your provider what its default is.
  • Blocking by IP can backfire. Anthropic warns that it may stop the crawler from reading your robots.txt at all.
  • Microsoft uses page tags. Bing announced new training and grounding preferences on October 30, 2026, set with meta tags addressed to Bingbot, and says they take effect from February 1, 2027. They replace the older noarchive and nocache tags for this purpose.

A shared "no AI training" signal for robots.txt is also being drafted at the IETF. Anthropic and Microsoft say they honor or will honor it, and Google says it intends to support it in early 2027. The draft may still change, so the named rules above remain the dependable route for now.

What llms.txt is, and its real status

llms.txt is a proposal published by Jeremy Howard in September 2024. It suggests placing a Markdown file at /llms.txt with your site's name, a short summary, and lists of links to your most useful pages, ideally in a clean text form. It is a guide for AI tools, not an access rule, so it does not replace robots.txt.

It is not dead either. OpenAI, Anthropic, Google's Gemini team and Perplexity publish llms.txt files for their own developer documentation, and Chrome's Lighthouse tool now checks for one, calling it an emerging convention that is optional for now. The practical use today is helping coding assistants and agents read documentation. If your site has docs or an API, the file is cheap to add. For a local business site, treat it as low priority.

Content that only appears with JavaScript

Many modern sites send a nearly empty page and then build the content in the visitor's browser with JavaScript. Google can run that JavaScript, in a separate rendering step that may come later. But Google's own documentation points out that not all bots can run JavaScript, and recommends server-side rendering or pre-rendering.

The OpenAI, Anthropic and Perplexity crawler pages do not say whether their crawlers execute scripts. The safe assumption is that they may not. To check, open your page, choose "View page source", and search for your prices, services and contact details. If they are missing from the source, ask your developer to render them on the server.

A sensible setup for a small site

  1. 1Read your current robots.txt and look for old rules that block everything.
  2. 2Allow the search crawlers and user-triggered fetchers of the engines your customers use.
  3. 3Decide about training crawlers as a business choice, and write the rule either way.
  4. 4Ask your host or CDN whether it blocks AI bots by default.
  5. 5Confirm your key facts are in the page source, not only added by JavaScript.
  6. 6Add llms.txt only if you have documentation worth pointing to.
  7. 7Recheck the official crawler pages a few times a year.

Rastock AEO, which is still in build, is meant to track how answer engines describe a business. None of that works if crawlers cannot read your site, so this setup comes first.

Questions and answers

Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI says each setting is independent: GPTBot relates to training, and OAI-SearchBot controls whether a site is shown in ChatGPT search answers.
Do I need an llms.txt file?
It is optional. It is a community proposal, Google Search says it does not use it, and we found no major engine confirming it affects answers. It is most useful for sites with developer documentation.
Do all AI bots obey robots.txt?
No. The file is a request that well-behaved crawlers follow. Perplexity says its user-triggered fetcher generally ignores it, and OpenAI says the rules may not apply to ChatGPT-User.
Will AI crawlers see content that my site loads with JavaScript?
Possibly not. Google renders JavaScript but notes that not all bots can, and the other major crawler pages do not say. Put important facts in the server-sent HTML.

Sources

  1. Overview of OpenAI Crawlers (OpenAI)
  2. Does Anthropic crawl data from the web, and how can site owners block the crawler? (Claude Help Center)
  3. Perplexity Crawlers (Perplexity)
  4. Google's common crawlers (Google for Developers)
  5. Optimizing your website for generative AI features on Google Search (Google Search Central)
  6. Robots.txt Introduction and Guide (Google Search Central)
  7. Understand the JavaScript SEO basics (Google Search Central)
  8. Your content, your choice: clearer preferences for training and grounding (Microsoft Bing Blog)
  9. The /llms.txt file, v2 (llmstxt.org)
  10. llms.txt (Lighthouse, Chrome for Developers)
ShareLinkedInX

See what AI says about your business.

Get early access

More articles