How to block GPTBot and other AI crawlers
Every major AI company publishes the name its crawlers use in robots.txt, so blocking them takes a few lines. The decision that matters is which ones: blocking training crawlers keeps your content out of future models, while blocking AI search crawlers also keeps you out of their answers and citations.
Training crawlers vs search and user agents
AI companies now run several bots each, and they do different jobs:
- Training crawlers collect pages that may be used to train future models. Blocking them is an opt-out from training, and it doesn't affect whether you appear in any search engine.
- AI search crawlers build the index that AI search answers draw on, and decide which pages can be cited. Blocking them means those tools won't show or link to your pages.
- User-triggered agents fetch a page because a person asked an assistant about it, such as pasting your URL into ChatGPT. Their vendors say robots.txt may not apply to these, because a person, not a crawler, made the request.
The AI crawler list
These are the names to use in robots.txt, taken from each vendor's own documentation:
| Token | Company | What it's for |
|---|---|---|
GPTBot | OpenAI | Training. Blocking it tells OpenAI not to use your content to train its models. |
OAI-SearchBot | OpenAI | ChatGPT search. OpenAI says sites that block it won't be shown in ChatGPT search answers. |
ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user asks. Not an automatic crawler; robots.txt may not apply. |
ClaudeBot | Anthropic | Training. |
Claude-SearchBot | Anthropic | Indexing for Claude's search results. |
Claude-User | Anthropic | Fetches a page when a Claude user asks. |
Google-Extended | Not a crawler but a control token: whether content Google already crawls may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI. It doesn't affect Google Search. | |
PerplexityBot | Perplexity | Perplexity's search index. Perplexity says it isn't used to train models. |
Perplexity-User | Perplexity | Fetches pages users ask about; Perplexity says it generally ignores robots.txt. |
Applebot-Extended | Apple | A control token, like Google-Extended: whether content Applebot crawls may train Apple's models. Siri and Spotlight are unaffected. |
meta-externalagent | Meta | Training AI models and indexing content for Meta's products. |
CCBot | Common Crawl | A free, public web archive that is widely used to build AI training sets. |
New bots appear regularly. Your server logs show which ones actually visit; anything you don't recognise can be checked against the vendor's documentation.
The robots.txt rules
Block AI training, stay in AI search. This is the most common choice: your content isn't used for training, but ChatGPT, Claude and Perplexity can still find and cite you.
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: CCBot Disallow: /
Listing several User-agent lines above one set of rules applies the rules to all of them, as RFC 9309 allows.
Block every AI crawler, including AI search: add OAI-SearchBot, Claude-SearchBot and PerplexityBot to the list above.
Block only part of the site, for example a members area or paid content:
User-agent: GPTBot User-agent: ClaudeBot Disallow: /members/ Disallow: /premium/
GPTBot a group, it ignores your User-agent: * rules entirely, so repeat any general rules in its group that it should still obey.Never block Googlebot or Bingbot to stop AI use. That removes you from normal search, and Google's AI Overviews come from the Google Search index, not from Google-Extended. Our robots.txt vs noindex guide covers what robots.txt does and doesn't control. To write the file without syntax mistakes:
Should you block them?
It's a business decision rather than an SEO rule. Some points to weigh:
- Blocking training crawlers has no known effect on search rankings. Google says Google-Extended isn't a ranking signal, and Apple says the same of Applebot-Extended. The cost is that future models know less about you, which matters if you want assistants to describe your brand accurately.
- Blocking AI search crawlers removes you from AI answers that link to sources. For many sites those links are a growing source of visits, so think carefully before blocking these.
- Publishers, paywalled sites and creators whose content is the product often block training. Businesses that want to be recommended usually allow search bots and decide case by case on training.
- Server load is a separate reason. If a crawler is hammering your server, a block or rate limit at your CDN may be needed regardless.
What robots.txt can't do
- It's voluntary. The companies above say they follow it, but nothing forces a crawler to. Bots that ignore it need blocking at the server or CDN.
- It isn't retroactive. A block stops future crawling; content already collected isn't removed from existing models.
- It takes time. Crawlers cache robots.txt; OpenAI says changes take about 24 hours to apply, and Meta says up to 24 hours.
- User-triggered fetches may ignore it, as described above.
- It applies per host. Each subdomain needs its own robots.txt.
If you use Cloudflare, its AI bot settings (under Security settings) can block AI crawlers at the network level, by category: search, agent and training. Cloudflare has announced new defaults for new domains from 15 September 2026 that block training and agent bots on pages showing ads. If AI bots seem to be blocked when your robots.txt allows them, check these settings.
How to check your rules
A typo in a user-agent name or a missing slash silently fails. Test your live robots.txt, or a draft, against real URLs for each AI crawler:
The tester shows, for every URL and bot, whether it's allowed and the exact line that decides it.
Questions
Should I block GPTBot?
Block it if you don't want your content used to train OpenAI's models. It doesn't affect Google rankings or ChatGPT search; ChatGPT search uses OAI-SearchBot, which you can allow separately.
Does blocking GPTBot remove my site from ChatGPT?
No. ChatGPT search results come from OAI-SearchBot, and pages users paste into ChatGPT are fetched by ChatGPT-User. To leave ChatGPT search, block OAI-SearchBot.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended doesn't affect inclusion in Google Search, and AI Overviews are part of Search. Controlling snippets in AI Overviews works through nosnippet and related directives, which also affect normal results.
How do I block GPTBot in Cloudflare?
Either add the robots.txt rules above, or use Cloudflare's AI bot settings under Security settings to block AI crawlers by category at the network level. The network block works even on bots that ignore robots.txt.
Will blocking AI crawlers hurt my SEO?
Blocking training crawlers won't affect Google or Bing rankings. Blocking Googlebot or Bingbot would, so never block those to stop AI use.