What is Googlebot?
Short answer: Googlebot is Google's main crawler for Google Search. Google's AI features in Search, AI Overviews and AI Mode, use the same crawling and indexing, so there is no separate Google AI search crawler to allow or block. Blocking Googlebot removes you from Google Search and its AI answers alike.
Googlebot at a glance
- Run by
- Type
- Search crawler
- Used for
- Google Search, AI Overviews and AI Mode
- robots.txt name
-
Googlebot - Follows robots.txt
- Yes
- Last checked
- 18 November 2026
- User agent
-
Googlebot/2.1; +http://www.google.com/bot.htmlBoth Googlebot Smartphone and Googlebot Desktop include this in their user agent. Google's documentation lists the full strings for each.
What does Googlebot do?
Googlebot is the umbrella name for two crawlers: Googlebot Smartphone and Googlebot Desktop. Google indexes the mobile version of most sites, so most crawl requests come from the smartphone crawler.
Google's guidance for its AI experiences in Search says the technical requirements are the same as for Search in general: pages must not block Googlebot, must return an HTTP 200 status and must have indexable content. There is no special AI crawler or file to add.
What happens if you block Googlebot?
Blocking Googlebot removes your pages from Google Search, and with them from AI Overviews and AI Mode, which draw on the same index.
If your goal is only to keep content out of Gemini model training or Gemini apps, the control is Google-Extended, not Googlebot.
How do you allow or block Googlebot in robots.txt?
Does Googlebot follow robots.txt? Yes. Googlebot Smartphone and Googlebot Desktop both obey the Googlebot token, so robots.txt cannot block one without the other.
Add one of these groups to the robots.txt file at the root of your site. Several User-agent lines in a row share the rules below them, and a crawler follows only the most specific group that names it, so repeat any private paths in that group.
Block Googlebot (removes you from Google Search)
User-agent: Googlebot
Disallow: / Allow Googlebot, keep /admin/ private
User-agent: Googlebot
Allow: /
Disallow: /admin/ Should you block Googlebot?
Our view: Never block Googlebot unless you want to leave Google entirely. To manage AI use of your content in Gemini apps and Vertex AI, use the Google-Extended token instead.
How do AI Overviews pick pages?
AI Overviews and AI Mode can only show pages that are in Google's index and eligible to appear with a snippet. Snippet controls such as nosnippet and max-snippet apply to these features too. Being indexed is the entry ticket; whether your page is chosen depends on how well it answers the question and how much Google trusts it.
How do you verify a real Googlebot request?
Google says user agents are often faked and recommends confirming Googlebot with a reverse DNS lookup on the source IP, or by matching the IP against the ranges Google publishes. Its verification page links the current JSON files.
Related AI crawlers
-
Google-ExtendedGoogle · Control token Not a separate crawler. A token that controls whether Google may use your content to train Gemini models and to ground answers in Gemini apps. -
OAI-SearchBotOpenAI · Search crawler Indexes pages so ChatGPT search can find, show and link to them. -
BingbotMicrosoft · Search crawler Crawls the web for Bing, whose index also grounds Copilot answers. -
PerplexityBotPerplexity · Search crawler Indexes pages so Perplexity can show and link them in its answers.
Readable is not the same as recommended: see whether Google's AI Overviews names your brand with the Google AI Overviews checker. See every AI crawler in one table, check your own robots.txt with the free AI crawler checker, or give AI tools a clean summary of your site with the llms.txt generator.
Googlebot questions
What is the Googlebot user agent?
Does Googlebot follow robots.txt?
Is there a separate crawler for Google AI Overviews?
Can I block AI Overviews without leaving Google Search?
Does blocking Google-Extended affect Googlebot?
How do I check whether my site allows Googlebot?
Crawlers can read you. Does AI recommend you?
See your ONE Score first · 7 day free trial, no card needed · Pay only if you keep tracking