Skip to content
Search crawler Run by Google

What is Googlebot?

Short answer: Googlebot is Google's main crawler for Google Search. Google's AI features in Search, AI Overviews and AI Mode, use the same crawling and indexing, so there is no separate Google AI search crawler to allow or block. Blocking Googlebot removes you from Google Search and its AI answers alike.

Googlebot at a glance

Run by
Google
Type
Search crawler
Used for
Google Search, AI Overviews and AI Mode
robots.txt name
Googlebot
Follows robots.txt
Yes
Last checked
18 November 2026
User agent
Googlebot/2.1; +http://www.google.com/bot.html Both Googlebot Smartphone and Googlebot Desktop include this in their user agent. Google's documentation lists the full strings for each.

What does Googlebot do?

Googlebot is the umbrella name for two crawlers: Googlebot Smartphone and Googlebot Desktop. Google indexes the mobile version of most sites, so most crawl requests come from the smartphone crawler.

Google's guidance for its AI experiences in Search says the technical requirements are the same as for Search in general: pages must not block Googlebot, must return an HTTP 200 status and must have indexable content. There is no special AI crawler or file to add.

What happens if you block Googlebot?

Blocking Googlebot removes your pages from Google Search, and with them from AI Overviews and AI Mode, which draw on the same index.

If your goal is only to keep content out of Gemini model training or Gemini apps, the control is Google-Extended, not Googlebot.

How do you allow or block Googlebot in robots.txt?

Does Googlebot follow robots.txt? Yes. Googlebot Smartphone and Googlebot Desktop both obey the Googlebot token, so robots.txt cannot block one without the other.

Add one of these groups to the robots.txt file at the root of your site. Several User-agent lines in a row share the rules below them, and a crawler follows only the most specific group that names it, so repeat any private paths in that group.

Block Googlebot (removes you from Google Search)

User-agent: Googlebot
Disallow: /

Allow Googlebot, keep /admin/ private

User-agent: Googlebot
Allow: /
Disallow: /admin/

Should you block Googlebot?

Our view: Never block Googlebot unless you want to leave Google entirely. To manage AI use of your content in Gemini apps and Vertex AI, use the Google-Extended token instead.

How do AI Overviews pick pages?

AI Overviews and AI Mode can only show pages that are in Google's index and eligible to appear with a snippet. Snippet controls such as nosnippet and max-snippet apply to these features too. Being indexed is the entry ticket; whether your page is chosen depends on how well it answers the question and how much Google trusts it.

How do you verify a real Googlebot request?

Google says user agents are often faked and recommends confirming Googlebot with a reverse DNS lookup on the source IP, or by matching the IP against the ranges Google publishes. Its verification page links the current JSON files.

Googlebot questions

What is the Googlebot user agent?
Requests contain Googlebot/2.1; +http://www.google.com/bot.html in the user agent. Both Googlebot Smartphone and Googlebot Desktop include this in their user agent. Google's documentation lists the full strings for each.
Does Googlebot follow robots.txt?
Yes. Googlebot Smartphone and Googlebot Desktop both obey the Googlebot token, so robots.txt cannot block one without the other.
Is there a separate crawler for Google AI Overviews?
No. AI Overviews and AI Mode use Google Search's normal crawling and index, so Googlebot is the crawler that matters.
Can I block AI Overviews without leaving Google Search?
Not with robots.txt. Snippet controls such as nosnippet limit how your content is shown, including in AI features, but blocking Googlebot removes you from Search entirely.
Does blocking Google-Extended affect Googlebot?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal.
How do I check whether my site allows Googlebot?
Run your domain through the free ONEGO AI crawler checker. It reads your robots.txt and shows whether Googlebot and 15 other AI crawlers are allowed, blocked or partly restricted, and what each rule means.

Crawlers can read you. Does AI recommend you?

See your ONE Score first · 7 day free trial, no card needed · Pay only if you keep tracking