A Shared ‘No’ Needs Its Own Line

Three of the largest companies that train AI models on web content said on October 30 that they would honor the same two lines in a website’s robots.txt file:

User-Agent: __usage_ai-training
Disallow: /

A group written under that name states a preference against AI training; individual paths can be exempted with Allow. Google said it aims to support the control in 2027 alongside other AI providers, and its crawler documentation names early 2027. Microsoft’s Bing team wrote that it would begin applying the new preferences on February 1, 2027, as its crawler revisits pages. OpenAI said it would implement the signal “in the coming months” and called the arrangement a “voluntary effort.”

The entry is defined in an Internet-Draft posted the same day by Kevin Kelley, who is listed as working at Anthropic. It is an individual submission: the Internet Engineering Task Force has not adopted it, it has no formal standing, and it expires in May 2027 unless renewed. It takes its definition of training from the IETF’s AI Preferences working group: using content to change the learned parameters of a generative model. I found no Anthropic announcement setting a date of its own; the company’s crawler help page, dated April 7, 2026, describes only its company-specific ClaudeBot block.

What the companies are offering is a single vocabulary where there were several. Since the fall of 2023, when Google introduced Google-Extended and Microsoft repurposed its NOARCHIVE and NOCACHE page tags, a site that wanted to keep its pages out of model training has had to learn each company’s word. OpenAI’s documentation says “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models”; Anthropic asks sites to restrict ClaudeBot. Microsoft’s 2023 rules were finer still: NOARCHIVE kept content out of its foundation-model training, while NOCACHE allowed only URLs, titles and snippets to be used. A site that missed a name was silent toward that company. One shared entry, if the companies keep their word, closes that gap.

The draft is exact about what qualifies as speaking. A wildcard group—User-Agent: *, the line many sites use to address every bot at once—must not be read as matching the new entry, and if no group names __usage_ai-training, “no preference has been expressed.” A blanket block remains what it always was, an instruction to crawlers; the draft leaves crawler rules untouched, and its token “is not a directive for crawling behavior.” But a site whose only robots.txt rule is a wildcard Disallow, written years ago to keep all bots out, has under this scheme said nothing about training. Keeping crawlers out and declining training are now two separate statements, and only the second is answered by the new entry.

How much that matters depends on the company. Google says “Existing company-specific controls will continue to function alongside this new joint signal,” and that a site using the shared entry without mentioning Google-Extended will be treated as having disallowed Google-Extended. OpenAI writes that its “existing crawler controls … will continue to apply and be respected.” Microsoft is moving further. It will stop reading NOARCHIVE and NOCACHE as preferences about training or about grounding—its term for feeding page content into generated answers, which it now treats as a separate choice—after March 1, 2027, or sooner on pages where its crawler finds the new preferences. “Both default to Allow when no applicable preference is published,” the Bing post says. A publisher that relied on NOARCHIVE to stay out of Microsoft’s model training will need to write the new preference to keep that position. Microsoft plans to open a temporary grounding setting in its webmaster tools in December; it has announced none for training.

The file the draft builds on has always worked by consent rather than force. RFC 9309, published in September 2022, specified and extended the method Martijn Koster proposed in 1994, and it states plainly that “These rules are not a form of access authorization.” The new draft keeps that character; its precedence section lets private agreements override preferences expressed this way. What has changed is the question the file is asked to answer. In 1994 it told a crawler where not to walk. From 2023, companies began reading it, alongside their own page tags and one name at a time, as a statement about what may be done with a page afterwards. The shared entry lets one sentence speak to all of them, and the draft decides that a sentence written for the old question does not answer the new one.

There is a fair case for that choice. A wildcard block written a decade ago to spare a small server from aggressive bots was not a statement about generative models, which few site operators were thinking about then; reading it as a refusal of training would put words in the operator’s mouth. The draft is careful not to guess what a site meant. It also stops at the moment of collection: the preferences that apply are those in the file “in effect when the asset was retrieved.” The draft does not address pages already gathered or models already trained on them, and I found no policy on either in the three companies’ announcements.

Care about intent is a virtue, but here it points one way. A site that has said nothing in the new vocabulary has, under the draft, expressed no preference, and at Microsoft that silence is written down as Allow. For a model being trained, the gap between a “no” and nothing at all can decide whether a page remains a candidate for its training data. The work of closing that gap falls on whoever runs the site, including some who believed they had already said no. Large publishers will read a Bing post about a March 1 deadline. A hobbyist forum, a local museum, a retired teacher’s archive of class notes probably will not.

The companies have answered a real complaint: too many names, no common word. Whether they have answered it fairly will show in 2027, in two places the announcements do not yet cover—whether any of them reports how many sites have written the new entry, and whether a site that writes it can ever confirm the preference was kept.