What is Claude-SearchBot?
Short answer: Claude-SearchBot is Anthropic's search crawler. Anthropic says it navigates the web to improve search result quality for Claude users, and that blocking it prevents your content being indexed for search, which may reduce your site's visibility and accuracy in Claude's search results.
Claude-SearchBot at a glance
- Run by
- Anthropic
- Type
- Search crawler
- Used for
- Claude's web search
- robots.txt name
-
Claude-SearchBot - Follows robots.txt
- Yes
- Last checked
- 18 November 2026
- User agent
-
Claude-SearchBotAnthropic names the bot but does not list a full user agent string on its help page.
What does Claude-SearchBot do?
Anthropic says Claude-SearchBot navigates the web to improve search result quality for users, analysing online content specifically to make Claude's search responses more relevant and accurate.
It is one of three Anthropic agents with separate controls: ClaudeBot for training, Claude-User for pages users ask about, and Claude-SearchBot for search.
What happens if you block Claude-SearchBot?
Anthropic says disabling Claude-SearchBot prevents its system from indexing your content for search optimisation, which may reduce your site's visibility and accuracy in user search results.
Blocking it does not opt you out of training (ClaudeBot) or stop Claude fetching a page a user asks for (Claude-User).
How do you allow or block Claude-SearchBot in robots.txt?
Does Claude-SearchBot follow robots.txt? Yes. Anthropic says its bots honour industry standard robots.txt directives, support the non standard Crawl-delay rule, and do not try to bypass CAPTCHAs. Add the rule to the robots.txt of every subdomain you want to cover.
Add one of these groups to the robots.txt file at the root of your site. Several User-agent lines in a row share the rules below them, and a crawler follows only the most specific group that names it, so repeat any private paths in that group.
Block Claude-SearchBot
User-agent: Claude-SearchBot
Disallow: / Allow Claude-SearchBot, keep /admin/ private
User-agent: Claude-SearchBot
Allow: /
Disallow: /admin/ Slow Claude-SearchBot down instead of blocking it
User-agent: Claude-SearchBot
Crawl-delay: 1 Should you block Claude-SearchBot?
Our view: Allow it if you want Claude to find and cite you when it searches the web. Blocking it trades away visibility without changing whether your content is used for training.
How do you verify a real Claude-SearchBot request?
Anthropic publishes the IP addresses its bots use at claude.com/crawling/bots.json (one list for all its bots). It advises against blocking those IP addresses to opt out, because the bots may then be unable to read your robots.txt.
Related AI crawlers
-
Claude-UserAnthropic · User fetcher Fetches a page when a Claude user asks it to. -
ClaudeBotAnthropic · Training crawler Collects content that may be used to train Anthropic's AI models. -
OAI-SearchBotOpenAI · Search crawler Indexes pages so ChatGPT search can find, show and link to them. -
GooglebotGoogle · Search crawler Crawls the web for Google Search, including AI Overviews and AI Mode.
See every AI crawler in one table, check your own robots.txt with the free AI crawler checker, or give AI tools a clean summary of your site with the llms.txt generator.
Claude-SearchBot questions
What is the Claude-SearchBot user agent?
Does Claude-SearchBot follow robots.txt?
Is Claude-SearchBot the same as ClaudeBot?
Does Claude-SearchBot support Crawl-delay?
What happens if I block Claude-SearchBot?
How do I check whether my site allows Claude-SearchBot?
Crawlers can read you. Does AI recommend you?
See your ONE Score first · 7 day free trial, no card needed · Pay only if you keep tracking