What is Google-Extended and does it affect AI Overviews?

Google-Extended is a robots.txt product token that tells Google whether content it crawls from your site may be used to train future Gemini models and to ground answers in Gemini Apps and Vertex AI. It does not affect AI Overviews. Google says the token does not change a site's inclusion or ranking in Google Search, and AI Overviews and AI Mode are part of Search. To limit what AI Overviews show from your pages, you use Googlebot rules and snippet controls instead.
What exactly is Google-Extended?
Google announced Google-Extended on 28 September 2023 as a control web publishers could use to manage whether their sites help improve Bard and Vertex AI, including future generations of their models (Google blog). Bard has since become Gemini.
Google's crawler documentation now describes it as a standalone product token covering two uses (Google Search Central):
Training: whether crawled content may be used to train future generations of Gemini models that power Gemini Apps and Vertex AI.
Grounding: whether content from Google's index may be provided to those models at prompt time.
Three details trip people up:
It is not a crawler. Google-Extended has no user agent string of its own. Crawling is done by Google's existing crawlers; the token only states how the content may be used.
You will not see it in your logs. Because it never requests pages, server logs cannot tell you whether it "visited".
It lives in robots.txt only. There is no meta tag version.
Does blocking Google-Extended affect AI Overviews or AI Mode?
No. Google's crawler page says Google-Extended "does not impact a site's inclusion in Google Search" and is not a ranking signal. Its guide to AI features adds that Google-Extended governs training and grounding in some of Google's other systems, not Search itself, and that crawling for Search, including AI Overviews and AI Mode, is controlled through Googlebot (Google Search Central).
In practice:
Block Google-Extended, keep Googlebot: you stay eligible for Search, AI Overviews and AI Mode.
Block Googlebot: you leave Search entirely, AI Overviews included. Rarely what anyone wants.
Peec AI's crawlability check classifies Google-Extended as a training token: an opt-out for Gemini training and AI product improvement (Peec AI docs).
What does Google-Extended affect, then?
Future Gemini model training. Blocking signals that your content should not be used.
Grounding in Gemini Apps and Vertex AI. Google's wording covers providing content from its Search index to these models at prompt time. Blocking may therefore limit how Gemini Apps use your pages when answering. Google does not publish figures on the size of that effect, so test it rather than assume.
That second point is the one most guides skip. For a publisher protecting paid content, limiting Gemini's use of it may be the goal. For a B2B brand that wants to be named in Gemini's answers, blocking may work against you.
How do I block or allow Google-Extended?
Add rules to your robots.txt at the root of the domain. To block the whole site:
User-agent: Google-Extended
Disallow: /To block only part of the site, for example an archive:
User-agent: Google-Extended
Disallow: /archive/To allow everything, either leave Google-Extended out (crawling follows your other rules) or add an explicit Allow: /.
Checks after you change it:
Fetch your live robots.txt and confirm the rule is there, on every host and subdomain you care about.
Make sure you did not edit the
Googlebotor*group by mistake. That is the change that hurts Search.Record the date. Any later change in Gemini visibility should be read against it.
How do I keep content out of AI Overviews if I want to?
Use Search controls, knowing they also change your normal results. Google lists these (Google Search Central):
nosnippet: no text snippet from the page in Search, including AI features.data-nosnippet: keeps a marked section of the page out of snippets.max-snippet: caps how much text may be shown.noindex: removes the page from the index altogether.
Google notes it can take several days to several months for changes to be recrawled, and that the URL Inspection tool shows the HTML Googlebot received. Semrush's guide makes the same trade-off point: these controls can limit AI Overview exposure but also reduce regular search visibility (Semrush).
How does Google-Extended compare with other AI crawler controls?
Each AI company splits its controls differently. The useful distinction is training versus search versus live fetching.
Google: Googlebot covers Search, AI Overviews and AI Mode. Google-Extended covers Gemini training and grounding. GoogleOther is a generic crawler whose robots.txt rules affect no specific product, and Google-CloudVertexBot fetches only what site owners request for Vertex AI agents.
OpenAI: GPTBot collects training data, OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User fetches pages a user asked for. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers (OpenAI developer docs).
So Google-Extended is the rough equivalent of GPTBot, not of OAI-SearchBot. Peec AI's research on bot types makes the same split, and notes that blocking one bot type does not affect the others (Peec AI research).
How do I audit my robots.txt for AI crawlers?
A 20 minute review catches most problems:
List every host you serve: www, the bare domain, docs, blog and app subdomains. Each has its own robots.txt.
Read each file top to bottom. Note every
User-agentgroup and what it disallows.Check the wildcard group. A broad
DisallowunderUser-agent: *applies to every bot without its own group, AI crawlers included.Map each named AI token to its job: training (Google-Extended, GPTBot), search (OAI-SearchBot), or live fetching (ChatGPT-User).
Compare against intent. If the goal is to be cited, search and live-fetch bots should be allowed on marketing and documentation pages.
Test a few URLs with a robots.txt tester and with Search Console's URL Inspection for Googlebot.
Keep a change log with the date and reason for every edit.
Our technical SEO audit checklist for AI answer engines covers the rest of the crawl and render checks.
How can I tell whether a change made any difference?
Measure before and after, on the same questions. Pick 20 to 40 buyer questions and record, weekly for a month before the change, whether Gemini names or cites you and whether Google's AI Overviews cite you. Make the change, then keep recording for at least a month. If Gemini moves and AI Overviews do not, the change did what Google describes. If both move, look for another cause, such as a ranking change or a site migration in the same weeks.
Should a business block Google-Extended?
Decide by what your content is for.
Block it if your content is the product (paid research, licensed data, premium journalism) and you do not want it training models or answering questions inside Gemini.
Allow it if your content exists to win customers. Product pages, comparisons, documentation and pricing pages are meant to be found and quoted.
Split it if you have both: block the premium paths, allow marketing and documentation.
Whatever you choose, write it down with the reason and the date. Robots.txt rules are often copied from a template and forgotten.
Which mistakes do teams make with Google-Extended?
Blocking it to "get out of AI Overviews". It will not work. Use snippet controls or noindex.
Copying an "AI block list" that includes Googlebot or OAI-SearchBot. That removes you from Search or ChatGPT search, not just training.
Expecting to see it in logs. It has no user agent.
Forgetting subdomains. Each host has its own robots.txt.
Never checking the result. Track Gemini and Google's AI surfaces before and after the change.
Where does Nagent's DRIS fit?
We make Nagent. DRIS, its AI search and answer engine visibility agent, helps you see what your crawler rules are doing in practice.
AI crawler analytics from your hosting or CDN logs, once Nagent's team switches it on: which answer-engine bots fetch your pages, split into live fetches, search indexing and training crawls.
Answers across engines: Gemini, ChatGPT, Claude, Perplexity and Grok, plus Google's AI Overviews and AI Mode, so you can compare Gemini against Google Search after a robots.txt change.
Before and after on the same questions: the prompt set is fixed per account, and every figure carries its run count, window and engine, so a change in Gemini's answers after a robots.txt edit is read against the same prompts.
A site audit and fixes your team approves, with index and canonical rules always going to a separate senior approval.
What it does not do: edit your robots.txt for you. Crawler rules stay a decision for your team.
Frequently Asked Questions
Is Google-Extended a crawler?
No. It is a robots.txt product token with no user agent string of its own. Google's existing crawlers do the crawling.
Does blocking Google-Extended hurt my Google rankings?
No. Google says it is not a ranking signal and does not affect inclusion in Google Search.
What is GPTBot and should I block it?
GPTBot is OpenAI's training crawler. Blocking it opts your content out of model training but does not remove you from ChatGPT search, which uses OAI-SearchBot. Most businesses that want to be recommended leave both allowed.
Why are AI crawlers hitting my site so often?
Search and live-fetch bots request pages when people ask questions, and training bots crawl in bursts. Check which kind is visiting before you block anything; the live fetches are the ones closest to a citation.
See which AI crawlers read your pages and what the engines say about you: book a Nagent demo and ask about DRIS.