Skip to content

How to Block AI Crawlers Without Losing AI Search (2026)

How to Block AI Crawlers Without Losing AI Search (2026)

AI companies read your website every day. Some copy your pages to train their next model. Some index them so ChatGPT or Perplexity can quote you, with a link back. Others fetch a single page because a person just pasted your URL into a chat.

Most guides tell you to block "AI bots" in one go. That throws away the useful bots along with the ones you don't want. Block the wrong name and your site quietly drops out of AI answers, or, worse, out of Google itself.

This guide shows you how to block AI crawlers the smart way: keep your content out of training if you want to, stay visible in AI search, and check the result in two minutes. Every bot name below comes from the company's own documentation, except ByteDance's Bytespider: ByteDance publishes none, so that one comes from Cloudflare's crawler reports.

Last checked: November 18, 2026.

The short version

You want to... Do this What happens
Stay out of AI training, stay in AI answers Block the training bots only (file 1 below) ChatGPT, Claude and Perplexity can still find and cite you
Keep all AI away Block training, search and user bots (file 2) You disappear from AI answers too
Let everything in Change nothing (file 3) Maximum reach, no control over training
Stay in Google and Bing Never block Googlebot or Bingbot Blocking them removes you from search

Not sure what your site allows right now? Run it through our free AI Crawler Checker. It reads your robots.txt the way the bots do and shows the answer bot by bot.

The three kinds of AI bots

Every major AI company now runs separate bots for separate jobs. That split is what lets you say yes to one and no to another.

  • Training crawlers collect pages to train AI models. Blocking them keeps your writing out of future training sets.
  • AI search crawlers index pages so an assistant can find and cite you, often with a link. Blocking them removes you from those answers.
  • User-triggered fetchers open one page because someone asked about it in a chat. They don't crawl the web on their own.
Company Training AI search User-triggered
OpenAI (ChatGPT) GPTBot OAI-SearchBot ChatGPT-User
Anthropic (Claude) ClaudeBot Claude-SearchBot Claude-User
Perplexity (none listed) PerplexityBot Perplexity-User
Google (Gemini) Google-Extended Googlebot (Search, AI Overviews, AI Mode) (none)
Apple Applebot-Extended Applebot (none)
Meta meta-externalagent Meta-WebIndexer Meta-ExternalFetcher
Amazon Amazonbot Amzn-SearchBot Amzn-User
Common Crawl CCBot (a free web archive many AI developers use)
ByteDance Bytespider

Two names on that list trip people up:

  • Google-Extended and Applebot-Extended are not crawlers. They're labels you use in robots.txt. Google and Apple keep crawling with their normal bots, then check these labels before using your pages for AI training.
  • Old names still float around. Anthropic no longer lists anthropic-ai or Claude-Web, and Meta no longer lists FacebookBot. Blocking them does nothing today.

Check your site before you change anything

You may already block more than you think. Hosts, plugins and CDNs add AI rules for you, and Cloudflare has changed its AI bot defaults several times since 2025.

  1. Open the AI Crawler Checker and enter your domain.
  2. Read the table. Each bot shows Allowed, Allowed, some pages blocked or Blocked, and the headline sums up your policy (for example "AI search yes, AI training no").
  3. Watch for a warning about Googlebot or Bingbot. If you see one, fix it first: it costs you search traffic today.
  4. If you use Cloudflare, open AI Crawl Control in your dashboard. Its settings can block bots at the network level, whatever your robots.txt says.

How to block AI crawlers in robots.txt

Your robots.txt file lives at yourdomain.com/robots.txt. Edit it in your hosting file manager, your SEO plugin, or your site's code. If you upload it yourself, 644 is the usual permission for a normal web file; our chmod calculator explains the numbers. Then pick one of these three files.

File 1: AI search yes, AI training no

This is the choice most small sites make. Your pages stay out of training sets, but ChatGPT, Claude and Perplexity can still find and cite you.

# AI training: no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# Everyone else, AI search bots included: yes
User-agent: *
Disallow:

Several User-agent lines on top of one set of rules is valid: the rules apply to every name above them.

File 2: Block all AI bots

Use this if you don't want any AI tool to read or quote your site. Search engines still get in.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Meta-WebIndexer
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /

User-agent: *
Disallow:

File 3: Let every bot in

User-agent: *
Disallow:

An empty Disallow: means "nothing is off limits". Keep any rules you already have, such as Disallow: /admin.

The mistake that breaks most AI rules

A bot that finds a group with its own name follows only that group and ignores your User-agent: * rules. So if your * group hides /checkout/ or /admin/, and you then add a separate group for OAI-SearchBot, that bot can now crawl /checkout/. Copy any private paths into every named group you add.

The checker's generator avoids this trap: it only writes groups for bots you block completely, so every other bot keeps following your * rules. Paste its lines above your existing User-agent: * group.

Quick poll

What will you do about AI crawlers on your site?

Tap an answer to see how other readers voted.

What about Google AI Overviews?

This is where most guides go wrong. Google uses three separate controls, and only one of them touches AI Overviews.

Control What it changes What it doesn't change
Block Google-Extended Gemini training, and Gemini grounding in Google's other AI products Google Search, rankings, AI Overviews, AI Mode
nosnippet, max-snippet or noindex on a page What Google can show from that page, in AI Overviews and AI Mode too Crawling
Block Googlebot Removes you from Google Search entirely, AI features included

Google says it plainly: Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". To limit what AI Overviews show from a page, Google points you to the snippet controls and noindex instead. It has also said it's exploring a dedicated opt-out for its generative AI features, so check its documentation before you rely on a guide that promises one.

Bing is moving too. On October 30, 2026, it announced separate preferences for AI training and AI grounding, which take effect on February 1, 2027. Neither affects your place in Bing search.

Does blocking AI crawlers hurt your SEO?

Not in Google or Bing, as long as you leave Googlebot and Bingbot alone. Google says Google-Extended isn't a ranking signal, and Apple says pages that block Applebot-Extended still show up in Spotlight, Siri and Safari.

The real trade-off is AI visibility. Block OAI-SearchBot and OpenAI says you "will not be shown in ChatGPT search answers". Block PerplexityBot and Perplexity can't cite you either. More people now find sites through these answers, so block search bots only if you've decided that traffic isn't worth it.

Should you block AI crawlers? A quick guide by site type

  • Blogs, tool sites and small businesses: use file 1. You want to be the source an assistant links to, and you lose little by opting out of training.
  • Paid content and news publishers: many block everything (see our survey below). When your content is the product, giving it away to AI tools for free rarely makes sense.
  • Online stores: let AI search in. Shoppers ask assistants which product to buy, and being left out can cost you sales.
  • Documentation and open-source projects: many let everything in. When AI tools know your product, they can help people use it.

What robots.txt can't do

  • It's a request, not a lock. The robots.txt standard (RFC 9309) says its rules "are not a form of access authorization". Big AI companies say they honour it. Scrapers that hide their name won't.
  • User-triggered fetchers may skip it. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules", because a person asked for the page. Anthropic says all its bots, Claude-User included, respect robots.txt.
  • It can't hide a page that's already public. For real protection, use a login or your host's bot blocking. Cloudflare's AI Crawl Control can block or charge AI bots at the network level.
  • llms.txt blocks nothing. It's a proposal for a file that guides AI tools around your site. Google says Search doesn't need it and ignores such files.

What 16 big sites do (our survey)

On November 18, 2026, we ran 20 well-known US, UK and Indian sites through our AI Crawler Checker. Four refused our request for their robots.txt. Here's what the other 16 tell each bot about their home page:

Site GPTBot ClaudeBot Google-Extended CCBot OAI-SearchBot PerplexityBot
The New York Times Blocked Blocked Blocked Blocked Blocked Blocked
BBC Blocked Blocked Blocked Blocked Blocked Blocked
Reddit Blocked Blocked Blocked Blocked Blocked Blocked
Amazon Blocked Blocked Blocked Blocked Blocked Blocked
CNN Blocked Blocked Partly Blocked Blocked Blocked
Reuters Blocked Blocked Blocked Blocked Partly Blocked
Quora Blocked Blocked Partly Partly Blocked Blocked
The Guardian Partly Blocked Partly Blocked Partly Blocked
The Washington Post Partly Blocked Partly Blocked Partly Blocked
Investopedia Partly Blocked Blocked Blocked Partly Blocked
Medium Blocked Blocked Partly Partly Partly Partly
Wikipedia Partly Partly Partly Partly Partly Partly
GitHub Partly Partly Partly Partly Partly Partly
Times of India Partly Partly Partly Partly Partly Partly
Mint Partly Partly Partly Partly Partly Partly
Hindustan Times Allowed Allowed Partly Allowed Allowed Allowed

"Partly" means the bot may read the home page, but some sections are off limits (the checker shows this as "Allowed, some pages blocked").

What stands out:

  • News sites in the US and UK block the most. The New York Times and the BBC block every AI bot we test.
  • Some AI names get blocked far more than others. Meta's meta-externalagent and ByteDance's Bytespider are fully blocked on 12 of the 16 sites and ClaudeBot on 11, but Google-Extended on only 6 and OAI-SearchBot on 6.
  • Indian news sites barely block anything. Times of India, Hindustan Times and Mint fully block only Meta's meta-externalagent.
  • Reddit shuts the door on everyone. Its robots.txt disallows every bot, Googlebot included.

These results show what each site's robots.txt said on the day we checked. Licensing deals and network-level blocking can change what bots actually get.

Quick answers

How do I block GPTBot in robots.txt? Add User-agent: GPTBot followed by Disallow: /. That stops OpenAI from using your pages for training. ChatGPT search uses OAI-SearchBot, so leave that one allowed if you want ChatGPT to cite you.

Does blocking ClaudeBot remove me from Claude? No. ClaudeBot collects training data. Claude's search uses Claude-SearchBot, and Claude-User fetches pages people ask about. Block only ClaudeBot to opt out of training.

Does blocking Google-Extended remove me from AI Overviews? No. Google says Google-Extended doesn't affect Search or AI Overviews. Use nosnippet or noindex on a page to limit what AI Overviews can show from it.

Can robots.txt actually stop AI crawlers? It stops the companies that follow it, which includes OpenAI, Anthropic, Google, Apple and Perplexity for their crawlers. It can't stop scrapers that ignore it. Use your host's or Cloudflare's bot blocking for those.

Does blocking AI bots hurt my Google rankings? No, unless you block Googlebot. AI-specific names like GPTBot, ClaudeBot and Google-Extended have no effect on Google Search.

Does llms.txt block AI crawlers? No. llms.txt is an optional guide for AI tools, not a set of rules. Use robots.txt to say what bots may crawl.

How do I check which AI bots can read my site? Enter your domain in our free AI Crawler Checker. It shows every major AI bot as allowed, allowed with some pages blocked, or blocked, and writes a robots.txt for the policy you pick.

Setting up a new server too? Our Linux VPS security checklist covers the rest of the basics.

Sources

The facts in this article were checked against these sources.

  1. OpenAI: Overview of OpenAI crawlers developers.openai.com
  2. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? support.claude.com
  3. Google: Google's common crawlers (Google-Extended) developers.google.com
  4. Google: AI features and your website developers.google.com
  5. Google: How Google interprets the robots.txt specification developers.google.com
  6. Google: Optimizing for generative AI features in Search developers.google.com
  7. Google: exploring controls for Search generative AI features blog.google
  8. Perplexity: Perplexity crawlers docs.perplexity.ai
  9. Apple: About Applebot support.apple.com
  10. Meta: Meta web crawlers developers.facebook.com
  11. Amazon: Amazonbot and Amazon crawlers developer.amazon.com
  12. Common Crawl: CCBot commoncrawl.org
  13. IETF: RFC 9309, Robots Exclusion Protocol rfc-editor.org
  14. Cloudflare: Block AI bots developers.cloudflare.com
  15. Cloudflare: Declaring your AIndependence (Bytespider) blog.cloudflare.com
  16. Bing: Clearer preferences for training and grounding (Oct 30, 2026) blogs.bing.com
  17. llms.txt proposal llmstxt.org