What is ClaudeBot?
Short answer: ClaudeBot is Anthropic's training crawler. Anthropic says it collects web content that could contribute to training its generative AI models, and that restricting ClaudeBot signals your future materials should be excluded from its training datasets. It does not control Claude's search, which uses Claude-SearchBot.
ClaudeBot at a glance
- Run by
- Anthropic
- Type
- Training crawler
- Used for
- Training Anthropic's models
- robots.txt name
-
ClaudeBot - Follows robots.txt
- Yes
- Last checked
- 18 November 2026
- User agent
-
ClaudeBotAnthropic names the bot but does not list a full user agent string on its help page.
What does ClaudeBot do?
Anthropic says ClaudeBot helps enhance the utility and safety of its generative AI models by collecting web content that could potentially contribute to their training.
Anthropic supports Crawl-delay, so you can slow ClaudeBot down instead of blocking it, for example with Crawl-delay: 1 under User-agent: ClaudeBot.
What happens if you block ClaudeBot?
Anthropic says that when a site restricts ClaudeBot, it signals that the site's future materials should be excluded from its AI model training datasets.
Blocking ClaudeBot does not remove you from Claude's search (Claude-SearchBot) or stop Claude reading a page a user asks about (Claude-User).
How do you allow or block ClaudeBot in robots.txt?
Does ClaudeBot follow robots.txt? Yes. Anthropic says its bots honour industry standard robots.txt directives, support the non standard Crawl-delay rule, and do not try to bypass CAPTCHAs. Add the rule to the robots.txt of every subdomain you want to cover.
Add one of these groups to the robots.txt file at the root of your site. Several User-agent lines in a row share the rules below them, and a crawler follows only the most specific group that names it, so repeat any private paths in that group.
Block ClaudeBot
User-agent: ClaudeBot
Disallow: / Allow ClaudeBot, keep /admin/ private
User-agent: ClaudeBot
Allow: /
Disallow: /admin/ Slow ClaudeBot down instead of blocking it
User-agent: ClaudeBot
Crawl-delay: 1 Should you block ClaudeBot?
Our view: A business choice, like GPTBot. If server load is the concern rather than training, use Crawl-delay instead of a full block. Keep Claude-SearchBot and Claude-User allowed if you want Claude to cite you.
What is Anthropic's AI training preference signal?
Anthropic says that as of 30 October 2026 it has begun recognising a machine readable signal, based on an IETF AI Preferences draft, that lets site owners say whether their content may be used to train AI models. It is rolling out across Anthropic's services over time. If your robots.txt names ClaudeBot, Anthropic follows that rule even if your general setting says something different.
How do you verify a real ClaudeBot request?
Anthropic publishes the IP addresses its bots use at claude.com/crawling/bots.json (one list for all its bots). It advises against blocking those IP addresses to opt out, because the bots may then be unable to read your robots.txt.
Related AI crawlers
-
Claude-SearchBotAnthropic · Search crawler Indexes pages to improve the search results Claude gives users. -
Claude-UserAnthropic · User fetcher Fetches a page when a Claude user asks it to. -
GPTBotOpenAI · Training crawler Collects content that may be used to train OpenAI's AI models. -
CCBotCommon Crawl · Training crawler Builds the open Common Crawl dataset, which many AI models are trained on.
See every AI crawler in one table, check your own robots.txt with the free AI crawler checker, or give AI tools a clean summary of your site with the llms.txt generator.
ClaudeBot questions
What is the ClaudeBot user agent?
Does ClaudeBot follow robots.txt?
Does blocking ClaudeBot remove me from Claude?
Can I slow ClaudeBot down instead of blocking it?
Should I block ClaudeBot by IP address?
How do I check whether my site allows ClaudeBot?
Crawlers can read you. Does AI recommend you?
See your ONE Score first · 7 day free trial, no card needed · Pay only if you keep tracking