AI Training Opt-Out in robots.txt: What Works Now
OpenAI and Google plan a shared AI-training opt-out. See what is live, how the draft token works, and the ChatGPT Search and Gemini visibility trade-offs.
On October 30, 2026, OpenAI and Google announced support plans for a shared robots.txt signal that lets website owners express a preference about AI model training across participating providers. It is a proposal and future rollout, not a switch that already blocks every AI system. The practical question for publishers is how to prepare without accidentally reducing search or assistant discovery. The answer starts with separating model training, web crawling, ChatGPT Search, and Gemini grounding: those are related but different uses of website content.
What OpenAI and Google actually announced
OpenAI says it intends to honor the shared training-preference signal in the coming months, while keeping its existing crawler controls. Google says it aims to support the mechanism in 2027. Neither announcement confirms universal adoption, a final Internet standard, or immediate enforcement across every AI provider. The underlying IETF document is an individual Internet-Draft, dated October 30, rather than an approved standard. Treat those distinctions as part of any implementation decision, especially if your business depends on organic search or AI-assisted referrals.
Announced rollout versus current controls
| Provider / document | Confirmed as of Nov 2, 2026 | What a site owner can rely on now |
|---|---|---|
| OpenAI | Plans to implement the shared signal in coming months; date not fixed | Existing documented GPTBot and OAI-SearchBot controls remain applicable |
| Plans support for shared training preference in 2027 | Existing Google-Extended controls and Google Search crawler documentation | |
| IETF individual draft | Defines a proposed policy token; not an adopted IETF standard | Technical proposal for evaluation, not universal provider enforcement |
What the proposed robots.txt token means
The draft names a policy token, __usage_ai-training. In its illustrative robots.txt group, User-agent: __usage_ai-training followed by Disallow: / expresses a preference against using site content to train generative AI models. A path-specific Allow rule can express a different preference for selected material. This is not a new HTTP crawler identity and does not, by itself, stop requests to your site. Providers must explicitly implement the policy interpretation before it affects their training decisions. The draft also says older parsers should continue behaving as before.
Publishing the proposed token today is not evidence that OpenAI, Google, Anthropic, Microsoft or every other provider already honors it. The draft may evolve, and website owners should not remove existing provider-specific protections on the assumption that one new group replaces them.
AI training and AI search visibility are different
A publisher may want to keep its pages available in search and assistant answers while withholding permission to use those pages for foundational model training. OpenAI's current crawler documentation separates GPTBot, which is associated with potential training use, from OAI-SearchBot, which supports ChatGPT search discovery. Blocking GPTBot does not automatically mean blocking OAI-SearchBot. ChatGPT-User is different again: it can fetch pages in response to a person's request, and ordinary robots.txt rules may not govern every user-triggered fetch.
Choose the control for the purpose
| Goal | Relevant documented control | Important caveat |
|---|---|---|
| Express an OpenAI training preference now | GPTBot rules in robots.txt | Do not assume the proposed shared token is already implemented |
| Remain eligible for ChatGPT Search discovery | Allow OAI-SearchBot to access eligible pages | Eligibility does not guarantee inclusion, citation or traffic |
| Manage Google's AI-training-related uses | Google-Extended rules | Google says explicit Google-Extended preferences may override the future shared default |
| Express a future shared training preference | Proposed __usage_ai-training policy-token group | Not a finalized standard or universal opt-out today |
What website owners should do now
- Inventory your current robots.txt file, CDN-managed rules and any provider-specific crawler policies. Check what each rule is intended to control rather than copying a generic all-bots block.
- Decide separately whether you want training opt-out, ordinary search indexing, ChatGPT Search eligibility and Gemini Apps grounding eligibility. Those goals are not interchangeable.
- Keep documented existing controls in place until the relevant provider confirms support for the new shared token. For OpenAI, review GPTBot and OAI-SearchBot independently.
- If testing the draft token, label it as experimental in your internal change log. Validate syntax, preserve existing rules and do not report it to stakeholders as an enforced universal restriction.
- Revisit official OpenAI, Google and IETF sources as implementation dates, precedence details and the draft itself change. Measure actual search indexing and referrals separately from crawler-policy configuration.
The Google-Extended and Gemini grounding trade-off
Google says that, once its planned shared control is supported, a broad AI-training opt-out may be interpreted like a Google-Extended disallow unless Google-Extended is explicitly specified. Google also says sites that explicitly allow Google-Extended are eligible to appear in grounded Gemini Apps answers and receive clicks. This is a future policy interaction, not proof that a newly added shared token already changes today's Gemini referrals. Owners who want both training restrictions and assistant visibility should evaluate that trade-off carefully before changing rules.
What remains unverified
- No final cross-provider IETF standard or universally binding implementation has been established by the cited announcements.
- OpenAI did not give a firm implementation day; Google describes a 2027 target rather than a live feature.
- Neither a robots.txt rule nor a useful article guarantees Google rankings, ChatGPT citations, or Gemini referrals.
- Other providers' exact adoption dates and precedence rules should not be inferred from OpenAI and Google's statements.
The most useful near-term change is operational, not technical: maintain a clear map of which bots and policies affect training versus discovery. A single shared signal could eventually simplify governance, but today a rushed switch could create a false sense of protection or an unintended visibility trade-off. For teams running content, SEO and AI workflows, version-controlled policy notes and periodic checks are more valuable than a one-time unverified opt-out claim.
Questions people ask
Is the new shared robots.txt AI-training opt-out working already?
No universal rollout has been confirmed as of November 2, 2026. OpenAI announced plans to honor the shared preference in the coming months, while Google targets 2027. The IETF document is still an individual draft, not an adopted standard. Continue using each provider's currently documented controls, and check their official announcements before assuming the new token is enforced.
If I block GPTBot, will my website disappear from ChatGPT Search?
Not automatically. OpenAI documents GPTBot for potential model-training crawls and OAI-SearchBot for ChatGPT Search discovery as independent controls. A site can express a training opt-out while allowing the search bot. That only preserves possible search eligibility; it does not promise that ChatGPT will retrieve, mention or cite a particular page. Review both rules in your robots.txt.
Could a future shared AI-training opt-out reduce Gemini Apps citations?
Potentially, depending on the Google-Extended setting. Google says that when it implements the shared signal, an opt-out without an explicit Google-Extended rule may be treated as a Google-Extended disallow. Google also links explicit Google-Extended permission with eligibility for grounded Gemini Apps answers. This is a planned interaction, not a verified current effect of the draft token.
What is __usage_ai-training, and is it an official IETF standard?
It is a proposed robots.txt policy token in an October 30, 2026 individual Internet-Draft. The draft illustrates a User-agent: __usage_ai-training group with Disallow or Allow paths to express a preference about generative model training. It does not create a new crawling bot or automatic enforcement. The IETF page explicitly says the document is not endorsed as a standard.
The shared AI-training signal is a meaningful direction for publisher choice, but it is not yet a universal live control. Keep GPTBot, OAI-SearchBot and Google-Extended decisions separate; use the proposed token only with its draft status clearly understood. Follow official implementation updates before replacing existing rules, and measure search or assistant discovery independently of training preferences.
Sources & useful resources
- OpenAI: Shared AI training preferences (Oct 30, 2026)— Future implementation; existing controls retained
- Google Search Central: Shared training control (Oct 30, 2026)— Google-Extended precedence, Gemini grounding and 2027 target
- IETF: Machine-Readable Training Preference Signal draft— Individual draft; proposed __usage_ai-training syntax; not adopted standard
- OpenAI developer docs: Overview of OpenAI Crawlers— GPTBot, OAI-SearchBot and ChatGPT-User separate roles