Three kinds of AI bots
Not all AI bots do the same thing, and the difference matters for your decision:
- Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta's crawler) collect pages to train AI models.
- AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index pages so assistants can find and cite you in answers, often with a link.
- User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) open a page because a person asked about it in a chat.
Many site owners now choose "AI search yes, AI training no": they stay visible in AI answers but keep their writing out of training data. The generator in the checker writes exactly those rules.
How the check works
We read your robots.txt the way crawlers do (RFC 9309): a bot follows the group that names it, and otherwise the * group. For each bot we test your home page and flag any other pages it is kept out of.
Watch out for one mistake: blocking Googlebot or Bingbot removes you from Google or Bing search, and Bing also feeds Copilot. The checker warns you if that happens.
robots.txt is a request, not a lock. Reputable companies follow it, but some bots ignore it or disguise themselves. For those, use your host's or Cloudflare's bot blocking.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
What is Google-Extended?
Is my site checked live?
Sources
The rules and figures in this tool come from these official sources, last checked November 13, 2026.
- RFC 9309: Robots Exclusion Protocol rfc-editor.org
- OpenAI: overview of OpenAI crawlers platform.openai.com
- Google: common crawlers developers.google.com
- Anthropic: how site owners can block the crawler support.claude.com
- Perplexity: crawlers docs.perplexity.ai