Skip to content

AI Crawler Checker

See which AI bots can read your website (ChatGPT, Claude, Perplexity, Gemini, Meta and more) and get a robots.txt that does what you want.

See which AI companies can read your site (ChatGPT, Claude, Perplexity, Gemini, Meta and more), and whether Google and Bing can still find you.

Three kinds of AI bots

Not all AI bots do the same thing, and the difference matters for your decision:

  • Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta's crawler) collect pages to train AI models.
  • AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index pages so assistants can find and cite you in answers, often with a link.
  • User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) open a page because a person asked about it in a chat.

Many site owners now choose "AI search yes, AI training no": they stay visible in AI answers but keep their writing out of training data. The generator in the checker writes exactly those rules.

How the check works

We read your robots.txt the way crawlers do (RFC 9309): a bot follows the group that names it, and otherwise the * group. For each bot we test your home page and flag any other pages it is kept out of.

Watch out for one mistake: blocking Googlebot or Bingbot removes you from Google or Bing search, and Bing also feeds Copilot. The checker warns you if that happens.

robots.txt is a request, not a lock. Reputable companies follow it, but some bots ignore it or disguise themselves. For those, use your host's or Cloudflare's bot blocking.

Frequently asked questions

Does blocking GPTBot remove me from ChatGPT?
Not from ChatGPT search: that uses OAI-SearchBot. Blocking GPTBot only stops OpenAI from using your pages for training.
What is Google-Extended?
It is not a separate crawler. It is a robots.txt name that tells Google whether pages Googlebot already fetched may be used for Gemini. Blocking it does not affect Google Search.
Is my site checked live?
Yes. We fetch your current robots.txt (results are cached for ten minutes) and store nothing else.

Sources

The rules and figures in this tool come from these official sources, last checked November 13, 2026.

  1. RFC 9309: Robots Exclusion Protocol rfc-editor.org
  2. OpenAI: overview of OpenAI crawlers platform.openai.com
  3. Google: common crawlers developers.google.com
  4. Anthropic: how site owners can block the crawler support.claude.com
  5. Perplexity: crawlers docs.perplexity.ai