AI crawlers list: every AI user agent and what it does
Short answer: AI companies read websites with three kinds of bots: search crawlers that index pages so AI tools can cite them, user fetchers that open a page when someone asks, and training crawlers that collect content for AI models. Allow the search crawlers and user fetchers if you want AI to recommend you; blocking training crawlers is optional and does not remove you from AI search.
Every AI crawler, who runs it and what blocking it means
Select a name for its user agent, robots.txt rules, verification method and sources.
| robots.txt name | Run by | Type | Follows robots.txt | If you block it |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Search crawler | Yes | ChatGPT search is less likely to show or cite your pages. |
| ChatGPT-User | OpenAI | User fetcher | Not always | OpenAI says robots.txt may not apply to these user requested visits. Where it does, ChatGPT may not be able to open your pages for users who ask. |
| Googlebot | Search crawler | Yes | Your pages drop out of Google Search and Google's AI answers. | |
| Google-Extended | Control token | Never visits | Gemini apps may not use your content. Google Search and AI Overviews are not affected. | |
| Bingbot | Microsoft | Search crawler | Yes | You lose visibility in Bing and in Copilot answers that draw on it. |
| PerplexityBot | Perplexity | Search crawler | Yes | Perplexity is less likely to cite your pages. |
| Perplexity-User | Perplexity | User fetcher | Generally no | Perplexity says this fetcher generally ignores robots.txt. A firewall block stops Perplexity reading your pages for live questions. |
| Claude-SearchBot | Anthropic | Search crawler | Yes | Claude's search is less likely to include your pages. |
| Claude-User | Anthropic | User fetcher | Yes | Claude may not be able to open your pages for users who ask. |
| Applebot | Apple | Search crawler | Yes | Siri and Spotlight are less likely to surface you. |
| DuckAssistBot | DuckDuckGo | Search crawler | Yes | DuckDuckGo's AI answers may not use your pages. |
| GPTBot | OpenAI | Training crawler | Yes | Your content is left out of future OpenAI training. ChatGPT search is controlled separately by OAI-SearchBot. |
| ClaudeBot | Anthropic | Training crawler | Yes | Your content is left out of future Anthropic training. Claude's search uses Claude-SearchBot. |
| Applebot-Extended | Apple | Control token | Never visits | Your content is left out of Apple model training. Siri and Spotlight are not affected. |
| CCBot | Common Crawl | Training crawler | Yes | Your content is left out of future Common Crawl datasets. |
| Meta-ExternalAgent | Meta | Training crawler | Yes | Your content is left out of Meta's AI training. |
What kinds of AI crawlers are there?
Four kinds, and each needs a different decision. Blocking the wrong one can quietly remove you from AI answers.
Search crawlers
Index pages in advance so an AI search tool can find, quote and link to them. Blocking one usually keeps you out of that tool's answers.
User fetchers
Visit a page live because a person asked the assistant something. Some operators say robots.txt may not apply to these visits.
Training crawlers
Collect content that may be used to train AI models. Blocking one is a business choice and does not remove you from AI search.
Control tokens
Never visit your site. They are robots.txt names the operator reads to decide how it may use content its other crawlers fetched.
Which AI crawlers should you allow?
Allow every crawler that decides whether AI tools can find and cite you, then make a separate, deliberate choice about training. The table groups the crawlers by what you want to achieve.
| If you want to | These crawlers matter | Note |
|---|---|---|
| Be found and cited in AI search | OAI-SearchBotPerplexityBotClaude-SearchBotGooglebotBingbotApplebotDuckAssistBot | These decide whether ChatGPT search, Perplexity, Claude, Google's AI Overviews and AI Mode, Copilot, Siri and DuckDuckGo can use your pages. |
| Let assistants open pages people ask about | ChatGPT-UserPerplexity-UserClaude-User | Live visits on behalf of a user. OpenAI and Perplexity say robots.txt may not apply to theirs. |
| Appear in Gemini app answers | Google-Extended | Also covers Gemini model training, so it is one decision for both. |
| Keep content out of AI training (optional) | GPTBotClaudeBotCCBotMeta-ExternalAgentApplebot-Extended | Block these if you prefer. None of them controls whether you appear in AI search. |
How do you block AI training but stay in AI search?
Give the search crawlers and user fetchers an Allow group, give the training crawlers a Disallow group, and leave everything else to your general rules. Several User-agent lines in a row share the rules below them, and each crawler follows only the most specific group that names it, so repeat private paths such as /admin/ in every group.
# AI search and answer crawlers: keep these allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: DuckAssistBot
Allow: /
Disallow: /admin/
# AI model training: opt out (optional)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Applebot-Extended
Disallow: /
# Everyone else, including Googlebot, Bingbot and Applebot
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xml Two decisions are left out on purpose. Google-Extended covers both Gemini training and Gemini app answers, so blocking it costs you visibility in Gemini apps; decide it on its own. And Applebot follows your Googlebot rules when it is not named, so check those too.
Which AI crawlers ignore robots.txt?
Fetchers that act for a user are the exception. Their operators say a person asked for the page, so robots.txt may not apply:
- ChatGPT-User: OpenAI says robots.txt rules may not apply because these actions are initiated by a user.
- Perplexity-User: Perplexity says it generally ignores robots.txt rules, since a user requested the fetch.
- Meta-ExternalFetcher: Meta says it may bypass robots.txt for fetches a user requested. It is separate from Meta-ExternalAgent.
Anthropic says its bots, including Claude-User, honour robots.txt. To stop a fetcher that ignores robots.txt, use a firewall or CDN rule matched to the operator's published IP addresses. The reverse problem is more common: a firewall or bot protection service blocking crawlers that robots.txt allows.
What changed for AI crawler controls in late 2026?
Several operators started moving from one robots.txt name per crawler toward shared signals that say how content may be used. Named rules still work everywhere, so keep them, and treat the new signals as an addition.
Google October 2026
Plans to support a shared AI training opt out in robots.txt, based on the IETF AI Preferences work. A site that opts out with it and does not name Google-Extended would be treated as disallowing Google-Extended. Google-Extended keeps working.
SourceMicrosoft October 2026
New AI Training and AI Grounding preferences replace NoArchive and NoCache for those uses, taking effect as Bing crawls from 1 February 2027. They control how content is used, not crawling.
SourceAnthropic 30 October 2026
Began recognising a machine readable AI training preference based on an IETF draft, rolling out over time. A robots.txt rule that names ClaudeBot still wins.
SourceIETF Drafts in progress
The AI Preferences working group's vocabulary and attachment drafts define labels such as train-ai and a Content-Usage rule. They are drafts, not standards, and the syntax has changed between versions.
SourceHow do you tell a real AI crawler from a fake one?
Check the source IP address, not just the user agent. User agent strings are easy to fake, and several operators warn that other bots impersonate theirs.
| Operator | How to verify |
|---|---|
| OpenAI | IP lists at openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json |
| Reverse DNS lookup, or the IP ranges linked from Google's crawler verification page | |
| Microsoft | The Verify Bingbot tool in Bing Webmaster Tools or at bing.com/toolbox/verify-bingbot |
| Perplexity | IP lists at perplexity.ai/perplexitybot.json and perplexity.ai/perplexity-user.json |
| Anthropic | One IP list for all its bots at claude.com/crawling/bots.json |
| Common Crawl | IP ranges at index.commoncrawl.org/ccbot.json, plus reverse DNS over IPv4 |
| Apple, DuckDuckGo, Meta | Each operator's crawler page, linked from the crawler's profile |
Other AI agents you may see in your logs
- Meta-ExternalFetcher: Meta's agent for fetches a user requests. Meta says it may bypass robots.txt.
- Google-CloudVertexBot: crawls sites whose owners asked for it when building Vertex AI Agents. Google says it does not affect Google Search or other products.
Both are documented on their operators' pages linked from the Meta-ExternalAgent and Google-Extended profiles. New agents appear often; the AI crawler checker shows how your robots.txt treats the main ones.
Related: the AI crawler definition in our glossary, the free llms.txt generator, and how to get your business found in AI search.
AI crawler questions
What is an AI crawler?
Which AI crawlers should I allow?
Does blocking GPTBot remove me from ChatGPT?
Is there a crawler for Google AI Overviews?
Do all AI crawlers obey robots.txt?
How often does this list change?
Crawlers can read you. Does AI recommend you?
See your ONE Score first · 7 day free trial, no card needed · Pay only if you keep tracking