OpenAI, Google, Microsoft, and Anthropic are backing a shared way for websites to say they don't want their content used to train generative AI. That's a meaningful change for anyone who publishes online. It also comes with a big asterisk: this is a preference signal, not a lock on your website.
The proposal arrives as an Internet-Draft, not a finished standard. It uses a file many website owners already know: robots.txt. The idea is to express a training preference once instead of keeping a separate rule for every AI company.
One signal, multiple companies
Here's the proposed site-wide opt-out:
User-Agent: __usage_ai-training
Disallow: /
Under the draft, that says the site operator prefers that content under those paths not be used for generative AI training. The proposal also supports path-specific rules, so a site can express different preferences for different sections.
That sounds simple because it is supposed to be. But don't confuse the new policy token with an actual bot. The draft says it is not a directive for crawling behavior. A company can still fetch a page for another purpose. If you need to manage traffic from a particular crawler, its own rules remain a separate question.
When will companies actually respect it?
OpenAI says it plans to implement the signal in the coming months and keep its existing crawler controls. Google's documentation says it intends to support the shared signal in early 2027. TechRadar reports that Anthropic has started implementation and Microsoft plans to begin honoring it on February 1, 2027.
Those are not identical commitments or a guarantee that every AI company will comply. The mechanism is voluntary, and an Internet-Draft can still change. Check each provider's current documentation before relying on it.
Training, search, and answers aren't the same thing
This distinction matters. The draft's token is about training a generative model. Crawling rules and other uses are separate. A provider's implementation can also have consequences beyond the name of the shared signal.
For example, Google's existing Google-Extended control covers both training future Gemini models and certain grounding uses, according to Google's documentation. TechRadar reports that Google plans to treat the shared opt-out like a Google-Extended disallow when the latter is not separately specified. That's a reason to read the provider-specific details rather than assume that every company draws the same line.
What site owners should do now
- Decide what you want. Training, search visibility, and appearances in AI answers are different decisions.
- Review existing controls. Don't remove crawler-specific or provider-specific rules just because a new shared signal has been proposed.
- Check implementation dates and interactions. Rollouts and definitions vary. Read the original documentation before editing a live file, and test changes carefully.
The bigger story isn't two lines in a text file. It's that the companies building these models are moving toward a common way to hear a publisher's preference. That's progress. Whether it becomes useful in practice depends on who implements it, what their controls cover, and whether they honor the choice.
Sources: IETF Internet-Draft · OpenAI · Google · TechRadar