OpenAI, Google, Microsoft and Anthropic to Support a Shared Way to Opt Out of AI Training, Letting Sites Signal Their Preference to All Four With One robots.txt Entry

On October 30, 2026 (US time), OpenAI, Google, Microsoft and Anthropic announced that they will support a mechanism that lets website operators state, through a shared robots.txt syntax, whether they want their site's content used to train generative AI. Until now, opting out of AI training required listing each company's crawler name, such as GPTBot or Google-Extended, one by one. With this change, a single entry will convey the preference to every company that supports it.

How it works: write "__usage_ai-training" in robots.txt

The basis is "Machine-Readable Training Preference Signal," a document that Anthropic's Kevin Kelley submitted the same day to the IETF (the body that discusses internet technical standards). In the "User-Agent" line of robots.txt, where a crawler name normally goes, you write the shared token "__usage_ai-training", and then use Disallow and Allow lines beneath it to indicate, path by path, whether content may be used for training.

The example in the document is as follows.

• User-Agent: __usage_ai-training
• Disallow: /
• Allow: /press-kit/

In this case, the site signals that it does not want any of its content used for training, while allowing training on content under /press-kit/.

The document also specifies the following points.

• "User-Agent: *" (the directive for all crawlers) does not apply to this token. To opt out of training, you must name this token explicitly
• "Training" here means use to change the parameters of a generative AI model. Models used only for classification or ranking are out of scope
• No new directives or syntax are added; the existing robots.txt syntax is used as is
• The meaning of entries that name a specific company's crawler, such as GPTBot, does not change

Note that this document is not an official IETF standard; it is at the stage of an individual proposal.

Each company's support

All four companies said they will support the shared token, while noting that their existing company-specific settings will continue to work. However, timing and handling differ by company.

• Anthropic: The company that submitted the document; it has reportedly already begun supporting it
• OpenAI: Will support it within the coming months. It says this does not replace its existing settings
• Microsoft (Bing): Will reflect this directive when crawling starting February 1, 2027
• Google: Aims to support it in 2027

Google's explanation includes a point that needs attention. If a site opts out of training with the shared token and says nothing about Google-Extended, Google plans to treat Google-Extended as disallowed as well. Google-Extended covers not only training of Gemini models but also whether content is used as grounding for answers in the Gemini app. Google says sites that explicitly allow Google-Extended will continue to appear as grounding for answers in the Gemini app and remain eligible to be clicked.

Bing also adds a setting to opt out of "grounding for answers"

On the same day, Microsoft also announced a setting, separate from training, that lets sites choose whether their pages are used as grounding for AI answers in Bing, Copilot and other products.

• Training: In addition to the shared robots.txt token, this can be specified per page with a meta tag (e.g., <meta name="Bingbot" content="ai-training:disallow">) or an HTTP header. Page-level settings take precedence over robots.txt settings
• Grounding for answers: Per page, you can choose "disallow," which refuses all use, or "disallowextended," which refuses only use outside Microsoft's own products

Microsoft says these settings do not change whether pages are eligible to appear in Bing's regular search results. At the same time, it explains that opting out of use as grounding makes pages less likely to appear in AI-generated answers. In addition, from March 1, 2027, it will no longer read "NoArchive" and "NoCache," which it has treated as AI-related preference signals, for that purpose. For the transition, from December 15, 2026, only sites using NoArchive or NoCache at that time will be able to set temporary grounding preferences in Bing Webmaster Tools.

CoAI's take

Until now, sites that wanted to opt out of AI training had to add a new name to robots.txt every time a new crawler appeared. With four major companies supporting the same syntax, that effort drops considerably. The spread of settings that treat generative AI training separately from use as grounding for AI answers is also a step forward for site operators.

That said, not everything will switch over right away. Google's and Microsoft's support starts in 2027, and OpenAI has said only "in the coming months" without giving a specific date. This mechanism also only communicates a preference; it has no power to make companies that do not support it comply. With Google, be aware that even if you only meant to opt out of training, your content will stop being used in Gemini app answers unless you allow Google-Extended. For the time being, it is safest to keep your existing company-specific entries as well.

Sources

• Machine-Readable Training Preference Signal (IETF Datatracker)
• Making it easier for websites to express AI training preferences (OpenAI)
• Supporting a shared AI training control for websites (Google Search Central Blog)
• Your content, your choice: clearer preferences for training and grounding (Microsoft Bing Blogs)
• Google, OpenAI, Microsoft, and Anthropic back a shared robots.txt AI training signal (Coywolf)