Skip to content
AI crawler directory

AI crawlers list: every AI user agent and what it does

Short answer: AI companies read websites with three kinds of bots: search crawlers that index pages so AI tools can cite them, user fetchers that open a page when someone asks, and training crawlers that collect content for AI models. Allow the search crawlers and user fetchers if you want AI to recommend you; blocking training crawlers is optional and does not remove you from AI search.

Check your robots.txt 16 crawlers and tokens · checked 18 November 2026

Every AI crawler, who runs it and what blocking it means

Select a name for its user agent, robots.txt rules, verification method and sources.

robots.txt name Run by Type Follows robots.txt If you block it
OAI-SearchBot OpenAI Search crawler Yes ChatGPT search is less likely to show or cite your pages.
ChatGPT-User OpenAI User fetcher Not always OpenAI says robots.txt may not apply to these user requested visits. Where it does, ChatGPT may not be able to open your pages for users who ask.
Googlebot Google Search crawler Yes Your pages drop out of Google Search and Google's AI answers.
Google-Extended Google Control token Never visits Gemini apps may not use your content. Google Search and AI Overviews are not affected.
Bingbot Microsoft Search crawler Yes You lose visibility in Bing and in Copilot answers that draw on it.
PerplexityBot Perplexity Search crawler Yes Perplexity is less likely to cite your pages.
Perplexity-User Perplexity User fetcher Generally no Perplexity says this fetcher generally ignores robots.txt. A firewall block stops Perplexity reading your pages for live questions.
Claude-SearchBot Anthropic Search crawler Yes Claude's search is less likely to include your pages.
Claude-User Anthropic User fetcher Yes Claude may not be able to open your pages for users who ask.
Applebot Apple Search crawler Yes Siri and Spotlight are less likely to surface you.
DuckAssistBot DuckDuckGo Search crawler Yes DuckDuckGo's AI answers may not use your pages.
GPTBot OpenAI Training crawler Yes Your content is left out of future OpenAI training. ChatGPT search is controlled separately by OAI-SearchBot.
ClaudeBot Anthropic Training crawler Yes Your content is left out of future Anthropic training. Claude's search uses Claude-SearchBot.
Applebot-Extended Apple Control token Never visits Your content is left out of Apple model training. Siri and Spotlight are not affected.
CCBot Common Crawl Training crawler Yes Your content is left out of future Common Crawl datasets.
Meta-ExternalAgent Meta Training crawler Yes Your content is left out of Meta's AI training.

What kinds of AI crawlers are there?

Four kinds, and each needs a different decision. Blocking the wrong one can quietly remove you from AI answers.

Search crawlers

Index pages in advance so an AI search tool can find, quote and link to them. Blocking one usually keeps you out of that tool's answers.

User fetchers

Visit a page live because a person asked the assistant something. Some operators say robots.txt may not apply to these visits.

Training crawlers

Collect content that may be used to train AI models. Blocking one is a business choice and does not remove you from AI search.

Control tokens

Never visit your site. They are robots.txt names the operator reads to decide how it may use content its other crawlers fetched.

Which AI crawlers should you allow?

Allow every crawler that decides whether AI tools can find and cite you, then make a separate, deliberate choice about training. The table groups the crawlers by what you want to achieve.

If you want to These crawlers matter Note
Be found and cited in AI search OAI-SearchBotPerplexityBotClaude-SearchBotGooglebotBingbotApplebotDuckAssistBot These decide whether ChatGPT search, Perplexity, Claude, Google's AI Overviews and AI Mode, Copilot, Siri and DuckDuckGo can use your pages.
Let assistants open pages people ask about ChatGPT-UserPerplexity-UserClaude-User Live visits on behalf of a user. OpenAI and Perplexity say robots.txt may not apply to theirs.
Appear in Gemini app answers Google-Extended Also covers Gemini model training, so it is one decision for both.
Keep content out of AI training (optional) GPTBotClaudeBotCCBotMeta-ExternalAgentApplebot-Extended Block these if you prefer. None of them controls whether you appear in AI search.

How do you block AI training but stay in AI search?

Give the search crawlers and user fetchers an Allow group, give the training crawlers a Disallow group, and leave everything else to your general rules. Several User-agent lines in a row share the rules below them, and each crawler follows only the most specific group that names it, so repeat private paths such as /admin/ in every group.

# AI search and answer crawlers: keep these allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: DuckAssistBot
Allow: /
Disallow: /admin/

# AI model training: opt out (optional)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including Googlebot, Bingbot and Applebot
User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Two decisions are left out on purpose. Google-Extended covers both Gemini training and Gemini app answers, so blocking it costs you visibility in Gemini apps; decide it on its own. And Applebot follows your Googlebot rules when it is not named, so check those too.

Which AI crawlers ignore robots.txt?

Fetchers that act for a user are the exception. Their operators say a person asked for the page, so robots.txt may not apply:

  • ChatGPT-User: OpenAI says robots.txt rules may not apply because these actions are initiated by a user.
  • Perplexity-User: Perplexity says it generally ignores robots.txt rules, since a user requested the fetch.
  • Meta-ExternalFetcher: Meta says it may bypass robots.txt for fetches a user requested. It is separate from Meta-ExternalAgent.

Anthropic says its bots, including Claude-User, honour robots.txt. To stop a fetcher that ignores robots.txt, use a firewall or CDN rule matched to the operator's published IP addresses. The reverse problem is more common: a firewall or bot protection service blocking crawlers that robots.txt allows.

What changed for AI crawler controls in late 2026?

Several operators started moving from one robots.txt name per crawler toward shared signals that say how content may be used. Named rules still work everywhere, so keep them, and treat the new signals as an addition.

Google October 2026

Plans to support a shared AI training opt out in robots.txt, based on the IETF AI Preferences work. A site that opts out with it and does not name Google-Extended would be treated as disallowing Google-Extended. Google-Extended keeps working.

Source

Microsoft October 2026

New AI Training and AI Grounding preferences replace NoArchive and NoCache for those uses, taking effect as Bing crawls from 1 February 2027. They control how content is used, not crawling.

Source

Anthropic 30 October 2026

Began recognising a machine readable AI training preference based on an IETF draft, rolling out over time. A robots.txt rule that names ClaudeBot still wins.

Source

IETF Drafts in progress

The AI Preferences working group's vocabulary and attachment drafts define labels such as train-ai and a Content-Usage rule. They are drafts, not standards, and the syntax has changed between versions.

Source

How do you tell a real AI crawler from a fake one?

Check the source IP address, not just the user agent. User agent strings are easy to fake, and several operators warn that other bots impersonate theirs.

Operator How to verify
OpenAI IP lists at openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json
Google Reverse DNS lookup, or the IP ranges linked from Google's crawler verification page
Microsoft The Verify Bingbot tool in Bing Webmaster Tools or at bing.com/toolbox/verify-bingbot
Perplexity IP lists at perplexity.ai/perplexitybot.json and perplexity.ai/perplexity-user.json
Anthropic One IP list for all its bots at claude.com/crawling/bots.json
Common Crawl IP ranges at index.commoncrawl.org/ccbot.json, plus reverse DNS over IPv4
Apple, DuckDuckGo, Meta Each operator's crawler page, linked from the crawler's profile

Other AI agents you may see in your logs

  • Meta-ExternalFetcher: Meta's agent for fetches a user requests. Meta says it may bypass robots.txt.
  • Google-CloudVertexBot: crawls sites whose owners asked for it when building Vertex AI Agents. Google says it does not affect Google Search or other products.

Both are documented on their operators' pages linked from the Meta-ExternalAgent and Google-Extended profiles. New agents appear often; the AI crawler checker shows how your robots.txt treats the main ones.

Related: the AI crawler definition in our glossary, the free llms.txt generator, and how to get your business found in AI search.

AI crawler questions

What is an AI crawler?
An AI crawler is a bot an AI company uses to read websites. Some index pages so AI search tools can cite them, some fetch a page live when a user asks, and some collect content to train AI models. Each has a name you can allow or block in robots.txt.
Which AI crawlers should I allow?
If you want AI to recommend you, allow the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot, Applebot and DuckAssistBot) and the user fetchers. Blocking training crawlers such as GPTBot, ClaudeBot and CCBot is optional and does not remove you from AI search.
Does blocking GPTBot remove me from ChatGPT?
No. OpenAI controls ChatGPT search with OAI-SearchBot and training with GPTBot, separately. You can block GPTBot and still appear in ChatGPT search answers.
Is there a crawler for Google AI Overviews?
No separate one. AI Overviews and AI Mode use Google Search's normal crawling with Googlebot. Google-Extended covers Gemini apps and training and, Google says, does not affect Search.
Do all AI crawlers obey robots.txt?
Not all of them. OpenAI says robots.txt may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, and Meta says Meta-ExternalFetcher may bypass it, because a user asked for the page. Anthropic says its bots honour robots.txt.
How often does this list change?
Operators add and rename crawlers regularly. This directory was checked against each operator's documentation on 18 November 2026, and each profile links to its sources.

Crawlers can read you. Does AI recommend you?

See your ONE Score first · 7 day free trial, no card needed · Pay only if you keep tracking