What is Google-Extended?
Short answer: Google-Extended is a robots.txt token, not a crawler. It controls whether content Google crawls may be used to train future Gemini models and to ground answers in Gemini apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search, so AI Overviews are not affected.
Google-Extended at a glance
- Run by
- Type
- Control token
- Used for
- Gemini apps and Vertex AI
- robots.txt name
-
Google-Extended - Follows robots.txt
- Control token, never visits
- Last checked
- 18 November 2026
- User agent
- None of its own Google says Google-Extended has no separate HTTP user agent. Crawling is done with existing Google user agents and the token is read only as a control.
What does Google-Extended do?
Google describes Google-Extended as a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini apps and the Vertex AI API for Gemini. It also covers grounding: using your pages to answer questions in Gemini apps and in Grounding with Google Search on Vertex AI.
A separate token, Google-CloudVertexBot, covers crawls that site owners request when building Vertex AI Agents. Google says it does not affect Google Search or other products.
What happens if you block Google-Extended?
Disallowing Google-Extended keeps your content out of future Gemini training and out of grounded answers in Gemini apps and Vertex AI. Google's October 2026 post notes that sites which specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini apps.
Google says using Google-Extended does not affect a site's inclusion in Search, and Google-Extended is not used as a ranking signal. AI Overviews and AI Mode are part of Search, so they are controlled by Googlebot, not by this token.
How do you set Google-Extended in robots.txt?
Is Google-Extended a crawler? It is a control, not a crawler. Google reads your Google-Extended rules when deciding how it may use content its other crawlers fetched; no request ever arrives from Google-Extended itself.
Add one of these groups to the robots.txt file at the root of your site. Several User-agent lines in a row share the rules below them, and a crawler follows only the most specific group that names it, so repeat any private paths in that group.
Opt out with Google-Extended
User-agent: Google-Extended
Disallow: / Allow Google-Extended
User-agent: Google-Extended
Allow: / Should you block Google-Extended?
Our view: If you want Gemini to cite and link to you, leave Google-Extended allowed. Blocking it is a reasonable choice for publishers who license content, but it costs you visibility in Gemini app answers.
What is changing with Google-Extended in 2027?
In October 2026 Google said it plans to support a shared AI training control in robots.txt, based on work in the IETF AI Preferences (AIPREF) working group. If a site uses that shared signal to opt out of training and does not name Google-Extended, Google plans to treat it as if the site had disallowed Google-Extended. Google says company specific controls such as Google-Extended will keep working alongside it.
How do you verify a real Google-Extended request?
There is nothing to verify, because Google-Extended never visits your site. Requests from Google come from its normal crawlers, which you can verify with Google's published IP ranges.
Related AI crawlers
-
GooglebotGoogle · Search crawler Crawls the web for Google Search, including AI Overviews and AI Mode. -
Applebot-ExtendedApple · Control token Not a separate crawler. A token that controls whether Apple may use content Applebot crawls to train its AI models.
Readable is not the same as recommended: see whether Gemini names your brand with the free Gemini visibility checker. See every AI crawler in one table, check your own robots.txt with the free AI crawler checker, or give AI tools a clean summary of your site with the llms.txt generator.
Google-Extended questions
What is the Google-Extended user agent?
Does Google-Extended follow robots.txt?
Is Google-Extended a crawler?
Does blocking Google-Extended remove me from AI Overviews?
Does Google-Extended affect Gemini answers?
How do I check whether my site allows Google-Extended?
Crawlers can read you. Does AI recommend you?
See your ONE Score first · 7 day free trial, no card needed · Pay only if you keep tracking