How to Block AI Crawlers Without Losing AI Search (2026)
AI companies read your website every day. Some copy your pages to train their next model. Some index them so ChatGPT or Perplexity can quote you, with a link back. Others fetch a single page because a person just pasted your URL into a chat.
Most guides tell you to block "AI bots" in one go. That throws away the useful bots along with the ones you don't want. Block the wrong name and your site quietly drops out of AI answers, or, worse, out of Google itself.
This guide shows you how to block AI crawlers the smart way: keep your content out of training if you want to, stay visible in AI search, and check the result in two minutes. Every bot name below comes from the company's own documentation, except ByteDance's Bytespider: ByteDance publishes none, so that one comes from Cloudflare's crawler reports.
Last checked: November 18, 2026.
The short version
| You want to... | Do this | What happens |
|---|---|---|
| Stay out of AI training, stay in AI answers | Block the training bots only (file 1 below) | ChatGPT, Claude and Perplexity can still find and cite you |
| Keep all AI away | Block training, search and user bots (file 2) | You disappear from AI answers too |
| Let everything in | Change nothing (file 3) | Maximum reach, no control over training |
| Stay in Google and Bing | Never block Googlebot or Bingbot |
Blocking them removes you from search |
Not sure what your site allows right now? Run it through our free AI Crawler Checker. It reads your robots.txt the way the bots do and shows the answer bot by bot.
The three kinds of AI bots
Every major AI company now runs separate bots for separate jobs. That split is what lets you say yes to one and no to another.
- Training crawlers collect pages to train AI models. Blocking them keeps your writing out of future training sets.
- AI search crawlers index pages so an assistant can find and cite you, often with a link. Blocking them removes you from those answers.
- User-triggered fetchers open one page because someone asked about it in a chat. They don't crawl the web on their own.
| Company | Training | AI search | User-triggered |
|---|---|---|---|
| OpenAI (ChatGPT) | GPTBot |
OAI-SearchBot |
ChatGPT-User |
| Anthropic (Claude) | ClaudeBot |
Claude-SearchBot |
Claude-User |
| Perplexity | (none listed) | PerplexityBot |
Perplexity-User |
| Google (Gemini) | Google-Extended |
Googlebot (Search, AI Overviews, AI Mode) |
(none) |
| Apple | Applebot-Extended |
Applebot |
(none) |
| Meta | meta-externalagent |
Meta-WebIndexer |
Meta-ExternalFetcher |
| Amazon | Amazonbot |
Amzn-SearchBot |
Amzn-User |
| Common Crawl | CCBot (a free web archive many AI developers use) |
||
| ByteDance | Bytespider |
Two names on that list trip people up:
Google-ExtendedandApplebot-Extendedare not crawlers. They're labels you use in robots.txt. Google and Apple keep crawling with their normal bots, then check these labels before using your pages for AI training.- Old names still float around. Anthropic no longer lists
anthropic-aiorClaude-Web, and Meta no longer listsFacebookBot. Blocking them does nothing today.
Check your site before you change anything
You may already block more than you think. Hosts, plugins and CDNs add AI rules for you, and Cloudflare has changed its AI bot defaults several times since 2025.
- Open the AI Crawler Checker and enter your domain.
- Read the table. Each bot shows Allowed, Allowed, some pages blocked or Blocked, and the headline sums up your policy (for example "AI search yes, AI training no").
- Watch for a warning about
GooglebotorBingbot. If you see one, fix it first: it costs you search traffic today. - If you use Cloudflare, open AI Crawl Control in your dashboard. Its settings can block bots at the network level, whatever your robots.txt says.
How to block AI crawlers in robots.txt
Your robots.txt file lives at yourdomain.com/robots.txt. Edit it in your hosting file manager, your SEO plugin, or your site's code. If you upload it yourself, 644 is the usual permission for a normal web file; our chmod calculator explains the numbers. Then pick one of these three files.
File 1: AI search yes, AI training no
This is the choice most small sites make. Your pages stay out of training sets, but ChatGPT, Claude and Perplexity can still find and cite you.
# AI training: no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /
# Everyone else, AI search bots included: yes
User-agent: *
Disallow:
Several User-agent lines on top of one set of rules is valid: the rules apply to every name above them.
File 2: Block all AI bots
Use this if you don't want any AI tool to read or quote your site. Search engines still get in.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Meta-WebIndexer
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /
User-agent: *
Disallow:
File 3: Let every bot in
User-agent: *
Disallow:
An empty Disallow: means "nothing is off limits". Keep any rules you already have, such as Disallow: /admin.
The mistake that breaks most AI rules
A bot that finds a group with its own name follows only that group and ignores your User-agent: * rules. So if your * group hides /checkout/ or /admin/, and you then add a separate group for OAI-SearchBot, that bot can now crawl /checkout/. Copy any private paths into every named group you add.
The checker's generator avoids this trap: it only writes groups for bots you block completely, so every other bot keeps following your * rules. Paste its lines above your existing User-agent: * group.
What about Google AI Overviews?
This is where most guides go wrong. Google uses three separate controls, and only one of them touches AI Overviews.
| Control | What it changes | What it doesn't change |
|---|---|---|
Block Google-Extended |
Gemini training, and Gemini grounding in Google's other AI products | Google Search, rankings, AI Overviews, AI Mode |
nosnippet, max-snippet or noindex on a page |
What Google can show from that page, in AI Overviews and AI Mode too | Crawling |
Block Googlebot |
Removes you from Google Search entirely, AI features included |
Google says it plainly: Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". To limit what AI Overviews show from a page, Google points you to the snippet controls and noindex instead. It has also said it's exploring a dedicated opt-out for its generative AI features, so check its documentation before you rely on a guide that promises one.
Bing is moving too. On October 30, 2026, it announced separate preferences for AI training and AI grounding, which take effect on February 1, 2027. Neither affects your place in Bing search.
Does blocking AI crawlers hurt your SEO?
Not in Google or Bing, as long as you leave Googlebot and Bingbot alone. Google says Google-Extended isn't a ranking signal, and Apple says pages that block Applebot-Extended still show up in Spotlight, Siri and Safari.
The real trade-off is AI visibility. Block OAI-SearchBot and OpenAI says you "will not be shown in ChatGPT search answers". Block PerplexityBot and Perplexity can't cite you either. More people now find sites through these answers, so block search bots only if you've decided that traffic isn't worth it.
Should you block AI crawlers? A quick guide by site type
- Blogs, tool sites and small businesses: use file 1. You want to be the source an assistant links to, and you lose little by opting out of training.
- Paid content and news publishers: many block everything (see our survey below). When your content is the product, giving it away to AI tools for free rarely makes sense.
- Online stores: let AI search in. Shoppers ask assistants which product to buy, and being left out can cost you sales.
- Documentation and open-source projects: many let everything in. When AI tools know your product, they can help people use it.
What robots.txt can't do
- It's a request, not a lock. The robots.txt standard (RFC 9309) says its rules "are not a form of access authorization". Big AI companies say they honour it. Scrapers that hide their name won't.
- User-triggered fetchers may skip it. OpenAI says robots.txt "may not apply" to
ChatGPT-User, and Perplexity saysPerplexity-User"generally ignores robots.txt rules", because a person asked for the page. Anthropic says all its bots,Claude-Userincluded, respect robots.txt. - It can't hide a page that's already public. For real protection, use a login or your host's bot blocking. Cloudflare's AI Crawl Control can block or charge AI bots at the network level.
- llms.txt blocks nothing. It's a proposal for a file that guides AI tools around your site. Google says Search doesn't need it and ignores such files.
What 16 big sites do (our survey)
On November 18, 2026, we ran 20 well-known US, UK and Indian sites through our AI Crawler Checker. Four refused our request for their robots.txt. Here's what the other 16 tell each bot about their home page:
| Site | GPTBot | ClaudeBot | Google-Extended | CCBot | OAI-SearchBot | PerplexityBot |
|---|---|---|---|---|---|---|
| The New York Times | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked |
| BBC | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked |
| Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | |
| Amazon | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked |
| CNN | Blocked | Blocked | Partly | Blocked | Blocked | Blocked |
| Reuters | Blocked | Blocked | Blocked | Blocked | Partly | Blocked |
| Quora | Blocked | Blocked | Partly | Partly | Blocked | Blocked |
| The Guardian | Partly | Blocked | Partly | Blocked | Partly | Blocked |
| The Washington Post | Partly | Blocked | Partly | Blocked | Partly | Blocked |
| Investopedia | Partly | Blocked | Blocked | Blocked | Partly | Blocked |
| Medium | Blocked | Blocked | Partly | Partly | Partly | Partly |
| Wikipedia | Partly | Partly | Partly | Partly | Partly | Partly |
| GitHub | Partly | Partly | Partly | Partly | Partly | Partly |
| Times of India | Partly | Partly | Partly | Partly | Partly | Partly |
| Mint | Partly | Partly | Partly | Partly | Partly | Partly |
| Hindustan Times | Allowed | Allowed | Partly | Allowed | Allowed | Allowed |
"Partly" means the bot may read the home page, but some sections are off limits (the checker shows this as "Allowed, some pages blocked").
What stands out:
- News sites in the US and UK block the most. The New York Times and the BBC block every AI bot we test.
- Some AI names get blocked far more than others. Meta's
meta-externalagentand ByteDance'sBytespiderare fully blocked on 12 of the 16 sites andClaudeBoton 11, butGoogle-Extendedon only 6 andOAI-SearchBoton 6. - Indian news sites barely block anything. Times of India, Hindustan Times and Mint fully block only Meta's
meta-externalagent. - Reddit shuts the door on everyone. Its robots.txt disallows every bot,
Googlebotincluded.
These results show what each site's robots.txt said on the day we checked. Licensing deals and network-level blocking can change what bots actually get.
Quick answers
How do I block GPTBot in robots.txt?
Add User-agent: GPTBot followed by Disallow: /. That stops OpenAI from using your pages for training. ChatGPT search uses OAI-SearchBot, so leave that one allowed if you want ChatGPT to cite you.
Does blocking ClaudeBot remove me from Claude?
No. ClaudeBot collects training data. Claude's search uses Claude-SearchBot, and Claude-User fetches pages people ask about. Block only ClaudeBot to opt out of training.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended doesn't affect Search or AI Overviews. Use nosnippet or noindex on a page to limit what AI Overviews can show from it.
Can robots.txt actually stop AI crawlers? It stops the companies that follow it, which includes OpenAI, Anthropic, Google, Apple and Perplexity for their crawlers. It can't stop scrapers that ignore it. Use your host's or Cloudflare's bot blocking for those.
Does blocking AI bots hurt my Google rankings?
No, unless you block Googlebot. AI-specific names like GPTBot, ClaudeBot and Google-Extended have no effect on Google Search.
Does llms.txt block AI crawlers? No. llms.txt is an optional guide for AI tools, not a set of rules. Use robots.txt to say what bots may crawl.
How do I check which AI bots can read my site? Enter your domain in our free AI Crawler Checker. It shows every major AI bot as allowed, allowed with some pages blocked, or blocked, and writes a robots.txt for the policy you pick.
Setting up a new server too? Our Linux VPS security checklist covers the rest of the basics.
Sources
The facts in this article were checked against these sources.
- OpenAI: Overview of OpenAI crawlers developers.openai.com
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? support.claude.com
- Google: Google's common crawlers (Google-Extended) developers.google.com
- Google: AI features and your website developers.google.com
- Google: How Google interprets the robots.txt specification developers.google.com
- Google: Optimizing for generative AI features in Search developers.google.com
- Google: exploring controls for Search generative AI features blog.google
- Perplexity: Perplexity crawlers docs.perplexity.ai
- Apple: About Applebot support.apple.com
- Meta: Meta web crawlers developers.facebook.com
- Amazon: Amazonbot and Amazon crawlers developer.amazon.com
- Common Crawl: CCBot commoncrawl.org
- IETF: RFC 9309, Robots Exclusion Protocol rfc-editor.org
- Cloudflare: Block AI bots developers.cloudflare.com
- Cloudflare: Declaring your AIndependence (Bytespider) blog.cloudflare.com
- Bing: Clearer preferences for training and grounding (Oct 30, 2026) blogs.bing.com
- llms.txt proposal llmstxt.org