Google plans shared AI-training controls: what publishers should prepare
Google targets 2027 support for a shared training preference. Publishers can audit existing controls now; the announcement is not a deployment guide.
Google plans to support a shared AI-training preference in robots.txt in 2027. For publishers reviewing crawler policies, the immediate task is to document existing choices and their consequences before introducing another signal.
The October 30 announcement describes a proposed shared policy token based on work at the Internet Engineering Task Force’s AIPREF working group. Google says provider-specific controls will continue alongside it. This plan is for future support, not a control publishers should assume already operates across providers.
How Google says the preferences would interact
Under the proposed approach, a training opt-out would be treated as a Google-Extended disallow when no explicit Google-Extended instruction exists. An explicit provider choice therefore matters when reviewing the broader preference. Google also says sites that specifically allow Google-Extended are eligible for grounding in Gemini Apps; eligibility does not promise that a page will be used or receive visits.
The announcement does not supply a production-ready rule to deploy. A proposed common preference also differs from agreement among AI providers on interpretation and timing. Keep the current Google crawler documentation beside the announcement when assessing a configuration.
A useful inventory before any configuration change
Consider a fictional publisher that wants ordinary search discovery but has separate decisions about model training and answer grounding. A single spreadsheet cell marked “AI allowed” cannot capture that policy. The owner needs to record the intended use, the named provider control, the current instruction, and who approved it.
For each entry, add the documentation URL, review date, and evidence needed to check the result. Keep a copy of the current robots.txt file and any relevant platform settings. An infrastructure rule that rejects requests and a published preference that asks a provider to limit a use act at different points; record both if both exist.
This is a proposed preparation exercise, not a test performed on a publisher’s site. It can expose conflicting intentions before a change, but it cannot establish whether a provider has enforced a future signal. Avoid treating a crawler request, an indexed URL, and an answer citation as equivalent evidence.
Keep the implementation decision tied to new evidence
Review the policy again when Google publishes supported syntax, an effective date, and instructions for conflicts between signals. Until then, use the Google-Extended guide for the existing documented control and the publisher bot-policy framework to keep each purpose explicit. No robots.txt change is justified solely by a future-support announcement.
Keep learning
Continue this topic
Next in this topic
Google plans Guided Learning pilot in Search for U.S. schools
Earlier in this topic
Commerce.com’s WebMCP rollout puts merchant opt-outs in the agent interface
AEO & AI Search
Ask a question or join the discussion