GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot: which AI robots to let in?
Each AI company sends several robots, for different purposes. Refusing them all on principle often means vanishing from answers without meaning to. Here is how to tell them apart.
Two families: answering and training
Some robots read your site to answer a question asked right now: refuse them and the assistant cannot quote you. Others read to build the training data of future models: refusing them is a legitimate choice, with no effect on today’s answers. The two carry similar names, hence the mistakes.
Those that read to answer:
OAI-SearchBotClaude-SearchBotClaude-UserPerplexityBotGoogle-Extended
Those that read to train:
GPTBotClaudeBotApplebot-ExtendedCCBot
Google-Extended stands apart: Google uses it to train Gemini, and also to ground its answers. Refusing it takes you out of Gemini’s answers; it does not touch your ranking in Google Search.
What they respect
Read again on 5 October 2026 in each company’s documentation: OAI-SearchBot, GPTBot, ClaudeBot, Claude-User, Claude-SearchBot and PerplexityBot respect your robots.txt. ChatGPT-User and Perplexity-User, which fetch a page because a user asked for it, say they ignore it: a block does not stop them.
A site that shuts the door without knowing
A robots.txt can refuse AI robots without the owner ever deciding it: a line added by a plugin, a theme, a contractor, or by a Cloudflare setting that writes rules against training robots itself. Since July 2025, Cloudflare also blocks AI robots by default at its firewall on new domains: that block cannot be read in robots.txt.
Each time we read your site, we read your robots.txt again, robot by robot. If the robots that read to answer are refused, we tell you at the top of the To improve page of your dashboard; if it is those that train, we note it as a possible choice, not a fault. A block at the firewall, invisible in the file, escapes us.
A rule that lets answers in and keeps training out
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /The robots that read to answer then fall under the general rule, which lets them in. Check that no other group in the file names them to refuse them.