Google-Extended Can Exclude Pages from Vertex AI Grounding Without Blocking Google Search
Use Google-Extended as a robots.txt control token for specified Gemini training and grounding uses while keeping Google Search crawling and ranking separate.
Direct answer: A publisher can disallow Google-Extended while continuing to allow Googlebot. Google documents Google-Extended as a standalone robots.txt control token for whether crawled content may be used for future Gemini training and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google also says the control does not affect inclusion or ranking in Google Search.
Google-Extended is not a separate request user agent, so you should not expect a matching user-agent string in server logs. It is also not documented as a universal opt-out from AI Overviews, AI Mode, every Gemini product, or owner-triggered Google-CloudVertexBot crawling.
Separate crawlers, tokens, and products
Policy mistakes start when a product name, robots token, and network crawler are treated as synonyms. Googlebot fetches for Search. Google-Extended supplies a use preference over content Google already crawls. Google-CloudVertexBot has a different, site-owner-requested Vertex AI Agent purpose. Record each as its own row with the official affected-products statement.
The July 14 documentation explicitly says Google-Extended is not a request UA. A server-log search for “Google-Extended” therefore cannot prove that the rule works or fails. The evidence is the robots.txt configuration, a successful fetch of that file, and the current Google documentation describing how the token is interpreted.
| Identity or control | What it governs | What not to claim |
|---|---|---|
| Googlebot | Google Search crawling | A Gemini training preference |
| Google-Extended | Named training and grounding uses | A separate request user agent |
| Google-CloudVertexBot | Owner-requested Vertex AI Agent crawling | Google Search inclusion |
| Search AI features | Normal Search eligibility and controls | Universally disabled by Google-Extended |
Write the smallest valid robots policy
Place the specific group in the site’s final, publicly accessible robots.txt and avoid duplicating contradictory groups. If the policy is to exclude the entire site from the named uses while leaving Search unchanged, a minimal group is easier to review than a long set of paths. If only an archive or licensed area is excluded, preserve a path inventory and test URL matching carefully.
Do not block Googlebot to express a Google-Extended preference. That would change Search crawling. Likewise, do not assume a Google-Extended rule controls user-triggered Vertex crawls, Gemini access to user-provided pages, or another vendor’s agents.
User-agent: Google-Extended
Disallow: /Verify the policy without a fake log test
Fetch robots.txt from several network locations, confirm HTTP 200 and the expected bytes, check redirects, CDN variants, staging overrides, encoding, capitalization, and deployment ownership. Save the before and after file, release ID, timestamp, reviewer, reason, rollback, and current source excerpt that names affected products.
Use a robots parser to test path matching, but label it a syntax and rule test—not proof of downstream product behavior. Because no Google-Extended request UA exists, avoid dashboard widgets that claim to count its hits. Monitor Googlebot and Search performance separately to detect accidental Search blocking.
Publish a precise control statement
A precise public statement says what token is configured, which paths it covers, which Google-documented uses it addresses, and the date checked. It should also say that Search inclusion and ranking are not affected by that token. Avoid broad promises such as “our site is excluded from Google AI.”
Review after Google updates the crawler registry, after a robots deployment, or when product names change. A legal or licensing policy may also require controls outside robots.txt, including access terms, authentication, and contractual restrictions.
Use the AI preview control guide for Search snippet rules and the AI crawler audit for non-Google controls.
Ask a question or join the discussion