Skip to content
Training crawler Run by Anthropic

What is ClaudeBot?

Short answer: ClaudeBot is Anthropic's training crawler. Anthropic says it collects web content that could contribute to training its generative AI models, and that restricting ClaudeBot signals your future materials should be excluded from its training datasets. It does not control Claude's search, which uses Claude-SearchBot.

ClaudeBot at a glance

Run by
Anthropic
Type
Training crawler
Used for
Training Anthropic's models
robots.txt name
ClaudeBot
Follows robots.txt
Yes
Last checked
18 November 2026
User agent
ClaudeBot Anthropic names the bot but does not list a full user agent string on its help page.

What does ClaudeBot do?

Anthropic says ClaudeBot helps enhance the utility and safety of its generative AI models by collecting web content that could potentially contribute to their training.

Anthropic supports Crawl-delay, so you can slow ClaudeBot down instead of blocking it, for example with Crawl-delay: 1 under User-agent: ClaudeBot.

What happens if you block ClaudeBot?

Anthropic says that when a site restricts ClaudeBot, it signals that the site's future materials should be excluded from its AI model training datasets.

Blocking ClaudeBot does not remove you from Claude's search (Claude-SearchBot) or stop Claude reading a page a user asks about (Claude-User).

How do you allow or block ClaudeBot in robots.txt?

Does ClaudeBot follow robots.txt? Yes. Anthropic says its bots honour industry standard robots.txt directives, support the non standard Crawl-delay rule, and do not try to bypass CAPTCHAs. Add the rule to the robots.txt of every subdomain you want to cover.

Add one of these groups to the robots.txt file at the root of your site. Several User-agent lines in a row share the rules below them, and a crawler follows only the most specific group that names it, so repeat any private paths in that group.

Block ClaudeBot

User-agent: ClaudeBot
Disallow: /

Allow ClaudeBot, keep /admin/ private

User-agent: ClaudeBot
Allow: /
Disallow: /admin/

Slow ClaudeBot down instead of blocking it

User-agent: ClaudeBot
Crawl-delay: 1

Should you block ClaudeBot?

Our view: A business choice, like GPTBot. If server load is the concern rather than training, use Crawl-delay instead of a full block. Keep Claude-SearchBot and Claude-User allowed if you want Claude to cite you.

What is Anthropic's AI training preference signal?

Anthropic says that as of 30 October 2026 it has begun recognising a machine readable signal, based on an IETF AI Preferences draft, that lets site owners say whether their content may be used to train AI models. It is rolling out across Anthropic's services over time. If your robots.txt names ClaudeBot, Anthropic follows that rule even if your general setting says something different.

How do you verify a real ClaudeBot request?

Anthropic publishes the IP addresses its bots use at claude.com/crawling/bots.json (one list for all its bots). It advises against blocking those IP addresses to opt out, because the bots may then be unable to read your robots.txt.

ClaudeBot questions

What is the ClaudeBot user agent?
Requests contain ClaudeBot in the user agent. Anthropic names the bot but does not list a full user agent string on its help page.
Does ClaudeBot follow robots.txt?
Yes. Anthropic says its bots honour industry standard robots.txt directives, support the non standard Crawl-delay rule, and do not try to bypass CAPTCHAs. Add the rule to the robots.txt of every subdomain you want to cover.
Does blocking ClaudeBot remove me from Claude?
Not from Claude's search. Search uses Claude-SearchBot and user requests use Claude-User, each with its own control.
Can I slow ClaudeBot down instead of blocking it?
Yes. Anthropic supports the non standard Crawl-delay rule, for example Crawl-delay: 1 in a ClaudeBot group.
Should I block ClaudeBot by IP address?
Anthropic advises against it, because ClaudeBot may then be unable to read your robots.txt, so the opt out may not work reliably. Use robots.txt.
How do I check whether my site allows ClaudeBot?
Run your domain through the free ONEGO AI crawler checker. It reads your robots.txt and shows whether ClaudeBot and 15 other AI crawlers are allowed, blocked or partly restricted, and what each rule means.

Crawlers can read you. Does AI recommend you?

See your ONE Score first · 7 day free trial, no card needed · Pay only if you keep tracking