Skip to content
Search crawler Run by Anthropic

What is Claude-SearchBot?

Short answer: Claude-SearchBot is Anthropic's search crawler. Anthropic says it navigates the web to improve search result quality for Claude users, and that blocking it prevents your content being indexed for search, which may reduce your site's visibility and accuracy in Claude's search results.

Claude-SearchBot at a glance

Run by
Anthropic
Type
Search crawler
Used for
Claude's web search
robots.txt name
Claude-SearchBot
Follows robots.txt
Yes
Last checked
18 November 2026
User agent
Claude-SearchBot Anthropic names the bot but does not list a full user agent string on its help page.

What does Claude-SearchBot do?

Anthropic says Claude-SearchBot navigates the web to improve search result quality for users, analysing online content specifically to make Claude's search responses more relevant and accurate.

It is one of three Anthropic agents with separate controls: ClaudeBot for training, Claude-User for pages users ask about, and Claude-SearchBot for search.

What happens if you block Claude-SearchBot?

Anthropic says disabling Claude-SearchBot prevents its system from indexing your content for search optimisation, which may reduce your site's visibility and accuracy in user search results.

Blocking it does not opt you out of training (ClaudeBot) or stop Claude fetching a page a user asks for (Claude-User).

How do you allow or block Claude-SearchBot in robots.txt?

Does Claude-SearchBot follow robots.txt? Yes. Anthropic says its bots honour industry standard robots.txt directives, support the non standard Crawl-delay rule, and do not try to bypass CAPTCHAs. Add the rule to the robots.txt of every subdomain you want to cover.

Add one of these groups to the robots.txt file at the root of your site. Several User-agent lines in a row share the rules below them, and a crawler follows only the most specific group that names it, so repeat any private paths in that group.

Block Claude-SearchBot

User-agent: Claude-SearchBot
Disallow: /

Allow Claude-SearchBot, keep /admin/ private

User-agent: Claude-SearchBot
Allow: /
Disallow: /admin/

Slow Claude-SearchBot down instead of blocking it

User-agent: Claude-SearchBot
Crawl-delay: 1

Should you block Claude-SearchBot?

Our view: Allow it if you want Claude to find and cite you when it searches the web. Blocking it trades away visibility without changing whether your content is used for training.

How do you verify a real Claude-SearchBot request?

Anthropic publishes the IP addresses its bots use at claude.com/crawling/bots.json (one list for all its bots). It advises against blocking those IP addresses to opt out, because the bots may then be unable to read your robots.txt.

Claude-SearchBot questions

What is the Claude-SearchBot user agent?
Requests contain Claude-SearchBot in the user agent. Anthropic names the bot but does not list a full user agent string on its help page.
Does Claude-SearchBot follow robots.txt?
Yes. Anthropic says its bots honour industry standard robots.txt directives, support the non standard Crawl-delay rule, and do not try to bypass CAPTCHAs. Add the rule to the robots.txt of every subdomain you want to cover.
Is Claude-SearchBot the same as ClaudeBot?
No. ClaudeBot collects content that may be used for training; Claude-SearchBot indexes content to improve Claude's search results. Each has its own robots.txt control.
Does Claude-SearchBot support Crawl-delay?
Yes. Anthropic says its bots support the non standard Crawl-delay extension to robots.txt.
What happens if I block Claude-SearchBot?
Anthropic says it may reduce your site's visibility and accuracy in Claude's search results.
How do I check whether my site allows Claude-SearchBot?
Run your domain through the free ONEGO AI crawler checker. It reads your robots.txt and shows whether Claude-SearchBot and 15 other AI crawlers are allowed, blocked or partly restricted, and what each rule means.

Crawlers can read you. Does AI recommend you?

See your ONE Score first · 7 day free trial, no card needed · Pay only if you keep tracking