Skip to content
Free tool

Free AI crawler checker: can AI read your site?

See whether ChatGPT, Gemini, Perplexity, Claude and other AI crawlers are allowed to read your website. We check your robots.txt against 16 AI crawlers and explain what each rule means for your AI visibility.

Free, no signup. We read your public robots.txt and llms.txt only.

AI search crawlers vs AI training crawlers

Not every AI crawler does the same job. Blocking the wrong one can quietly remove you from AI answers.

Search and answer crawlers: keep these allowed

These index your pages or fetch them live so AI tools can find, quote and link to you, for example OAI-SearchBot for ChatGPT search and PerplexityBot for Perplexity. If you want to be recommended in AI answers, they need to be able to read your site.

Training crawlers: your choice

These collect content that may be used to train AI models, for example GPTBot and CCBot. Blocking them is a legitimate business decision and does not by itself remove you from AI search, because search is handled by separate crawlers.

Every AI crawler we check, and what it does

The robots.txt name to use for each one, who runs it, and what blocking it means for you.

AI search and answer crawlers

robots.txt name Run by What it does If you block it
OAI-SearchBot OpenAI Indexes pages so ChatGPT search can find, show and link to them. ChatGPT search is less likely to show or cite your pages.
ChatGPT-User OpenAI Opens a page when a ChatGPT user asks it to, for example to read a link. ChatGPT may not be able to open your pages for users who ask.
Googlebot Google Crawls the web for Google Search, including AI Overviews and AI Mode. Your pages drop out of Google Search and Google's AI answers.
Google-Extended Google Not a separate crawler. A token that controls whether Google may use your content to train Gemini models and to ground answers in Gemini apps. Gemini apps may not use your content. Google Search and AI Overviews are not affected.
Bingbot Microsoft Crawls the web for Bing, which powers Copilot and feeds other AI search tools. You lose visibility in Bing, Copilot and tools that rely on Bing.
PerplexityBot Perplexity Indexes pages so Perplexity can show and link them in its answers. Perplexity is less likely to cite your pages.
Perplexity-User Perplexity Fetches a page when a Perplexity user's question needs it. Perplexity may not be able to read your pages for live questions.
Claude-SearchBot Anthropic Indexes pages to improve the search results Claude gives users. Claude's search is less likely to include your pages.
Claude-User Anthropic Fetches a page when a Claude user asks it to. Claude may not be able to open your pages for users who ask.
Applebot Apple Crawls the web for Siri and Spotlight suggestions. Siri and Spotlight are less likely to surface you.
DuckAssistBot DuckDuckGo Fetches pages for DuckDuckGo's AI assisted answers. DuckDuckGo's AI answers may not use your pages.

AI training crawlers

robots.txt name Run by What it does If you block it
GPTBot OpenAI Collects content that may be used to train OpenAI's AI models. Your content is left out of future OpenAI training. ChatGPT search is controlled separately by OAI-SearchBot.
ClaudeBot Anthropic Collects content that may be used to train Anthropic's AI models. Your content is left out of future Anthropic training. Claude's search uses Claude-SearchBot.
Applebot-Extended Apple Not a separate crawler. A token that controls whether Apple may use content Applebot crawls to train its AI models. Your content is left out of Apple model training. Siri and Spotlight are not affected.
CCBot Common Crawl Builds the open Common Crawl dataset, which many AI models are trained on. Your content is left out of future Common Crawl datasets.
Meta-ExternalAgent Meta Collects content to train Meta's AI models and improve its AI products. Your content is left out of Meta's AI training.

Crawler names and purposes are as published by each company and change over time. Check their documentation before making changes.

How to allow AI search crawlers in robots.txt

Your robots.txt file lives at the root of your site, for example example.com/robots.txt. Add a group for the AI crawlers you want to allow, and a separate group for any you want to opt out of.

  • Several User-agent lines in a row share the rules that follow them.
  • A crawler follows only the most specific group that names it, so repeat any private paths, such as /admin/, in every group.
  • The longest matching rule wins, and on a tie Allow beats Disallow.
  • Add a Sitemap line so crawlers can find every page.
# Let AI search and answer crawlers read the site
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
Allow: /
Disallow: /admin/

# Optional: opt out of AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

# Everyone else
User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

robots.txt is not the whole story

Being readable is the first step. Being recommended is the goal.

Firewalls and CDNs

Bot protection can block AI crawlers before they read robots.txt, and some providers can do this by default. Check your security settings too.

Content in the HTML

Many AI crawlers do not run JavaScript. Make sure your key content is in the page HTML, not only loaded in the browser.

Readable is not recommended

Crawlers reading your site is only the entry ticket. Whether AI actually names you depends on your content, facts and reputation.

Next step: see whether AI actually recommends you with our free AI visibility checker, or read how to get your business found in AI search.

AI crawler questions

What is an AI crawler?
An AI crawler is a bot that AI companies use to read websites. Some index pages so AI search tools such as ChatGPT search and Perplexity can find and cite them, some fetch a page when a user asks, and some collect content to train AI models. Each one has a name, called a user agent, that you can allow or block in robots.txt.
Does blocking GPTBot remove me from ChatGPT?
Not on its own. OpenAI says GPTBot is used for model training, while OAI-SearchBot is used for ChatGPT search and ChatGPT-User fetches pages when a user asks. Each is controlled separately, so you can block GPTBot and still allow OAI-SearchBot to keep showing up in ChatGPT search.
Should I block AI training crawlers?
That is a business choice. Blocking training crawlers such as GPTBot, ClaudeBot and CCBot keeps your content out of future model training, but it does not remove you from AI search. If you want to be recommended in AI answers, make sure the search and answer crawlers stay allowed.
What does Google-Extended do?
Google-Extended is not a separate crawler. It is a robots.txt token that controls whether Google may use content it crawls to train Gemini models and to ground answers in Gemini apps. Google says it does not affect Google Search, so blocking it does not remove you from AI Overviews, but it can keep you out of Gemini app answers.
My robots.txt allows AI crawlers. Can they definitely read my site?
Not always. robots.txt is a set of instructions, but a firewall, CDN or bot protection service can still block AI crawlers before they reach your pages, and some providers can block them by default. Check your security settings too, and make sure important content is in the page HTML rather than only loaded by JavaScript.
What is llms.txt?
llms.txt is an optional plain text file at the root of a website that summarises the site and links its key pages for AI tools. It is a proposed convention, Google has said it does not use it, and no major engine has confirmed it as a ranking signal, but it is quick to add and gives AI tools a clean summary.
How often should I check my AI crawler access?
Check whenever you change your robots.txt, your CDN or firewall settings, or your website platform, and every few months otherwise. New AI crawlers appear regularly, and a single misplaced Disallow rule can quietly remove you from AI answers.

Crawlers can read you. Does AI recommend you?

See your ONE Score first, no card needed · 3 day free trial after · Cancel anytime

Run your free AI audit