One robots.txt Line to Opt Out of AI Training: What Google, Microsoft, OpenAI and Anthropic Agreed To

On Friday, October 30, 2026, Google, Microsoft, OpenAI and Anthropic each said they will honor a single robots.txt signal that tells all of them, at once, not to use your content for AI training.

Until now you had to know every crawler by name. GPTBot, Google-Extended, ClaudeBot, CCBot, Applebot-Extended. A new lab meant a new line, and a forgotten line meant a silent yes.

That makes this a real change, even for a one-person blog. But the token covers training only. It does not stop crawling, does not remove you from search, and does not keep AI answers from citing you. Knowing where that line sits is the whole point of this post.

What happened

Kevin Kelley, Anthropic's technical lead for standards, submitted an IETF Internet-Draft, draft-kelley-aipref-training-preference-signal-00, dated October 30. The IETF is the body that maintains internet standards, including the robots.txt spec (RFC 9309).

The same day, Google published "Supporting a shared AI training control for websites" on its Search Central blog, Microsoft published "Your content, your choice" on the Bing blog, and OpenAI posted its own statement of support.

The signal looks like this:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

__usage_ai-training is not a real crawler. It is a "policy token" that reuses ordinary robots.txt syntax, so existing parsers already understand it. The example above means: no AI training on the whole site, except the press kit.

Rollout, as each company stated it (summarized by Engadget on October 31):

Background

OpenAI introduced GPTBot in August 2023 and Google followed with Google-Extended that September. For three years, opting out meant collecting crawler names.

Two problems were baked in. First, the list kept growing. Second, the names didn't do the same thing. GPTBot and ClaudeBot are actual crawlers, so blocking them reduces traffic. Google-Extended and Applebot-Extended are not crawlers at all. They only control how already-fetched content gets used. Same-looking lines, different effects.

The IETF's AIPREF (AI Preferences) working group has been sorting out vocabulary for this. Its vocab draft (draft-ietf-aipref-vocab-08) splits usage into train-ai, ai-use and search. A companion draft by Google's Gary Illyes and Mozilla's Martin Thomson proposes new syntax like Content-Usage: train-ai=n.

Kelley's draft borrows the working group's definition of training but skips the new syntax. One token, dropped into the robots.txt you already have. It reads as a bet that something usable today beats something perfect later.

What it blocks, and what it doesn't

The draft defines AI training, quoting the working group, as "using an asset to modify the learned parameters of a generative AI model." Training models for "exclusively non-generative tasks" is out of scope.

The limits are stated plainly:

• Crawling. The draft says the token "is not a directive for crawling behavior." If you want less bot traffic, you still need per-crawler rules or a WAF.
• Search. Google and Microsoft both say search results are unaffected.
• Grounding. When an AI product reads a page to answer a question and cites it, that's grounding. This token doesn't address it.
• Past collection. The preferences that apply are those in the robots.txt "in effect when the asset was retrieved." Nothing already trained gets untrained.
• Enforcement. Only companies that opt in will honor it. The draft also notes that private agreements may override the signal.

On the IETF mailing list, Kelley put it this way: this is a statement about content use, not a signal to control crawl traffic, and the two are deliberately separate.

Each company reads it a little differently

Google will treat the token as a Google-Extended disallow if you haven't set Google-Extended explicitly. But the same post says sites that "specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." So at Google, opting out of training appears to also cost you Gemini app citations.

Microsoft separates the two. Training is set via the robots.txt token or a page-level tag, <meta name="Bingbot" content="ai-training:disallow">. Grounding is page-level only, with three values: allow, disallowextended (Microsoft's own products only) and disallow. Microsoft also said that NoArchive and NoCache, which it has read as AI signals since 2023, stop meaning that on March 1, 2027. And it warns that restricting grounding "will significantly reduce your content's visibility in AI-generated experiences."

OpenAI called the effort voluntary and said it won't replace the controls it already offers. GPTBot (training) and OAI-SearchBot (ChatGPT search) remain independent switches.

Other views and open questions

The optimistic case is simple. Four companies backing one token on one day hasn't happened before, and it removes a chore that only got worse.

The caveats are real too. The draft is an individual -00 submission. The IETF page says it is "not endorsed by the IETF" and has "no formal standing." The syntax could change. Non-participating crawlers are untouched. Several outlets report Meta voiced support without committing. And because Google links training to Gemini grounding while Microsoft splits them, the outcome many publishers want, "don't train on me but do cite me," isn't one line away.

"AI can't take my content anymore" is the wrong takeaway. The accurate one: four companies promise that, starting around 2027, newly fetched content under this signal won't be used for training.

What this means for you

If you run a blog or small site

• Adding the token now costs nothing. Anthropic reads it already; the others start in 2027.
• Keep your existing GPTBot, Google-Extended and ClaudeBot lines through 2027. All three companies said existing controls continue to work.
• User-agent: * says nothing about training. The token has to be named.

If Bing sends you traffic

• If you relied on NoArchive or NoCache to stay out of Copilot, switch before March 1, 2027. A temporary transition setting opens in Bing Webmaster Tools on December 15, 2026.

If you're a developer

• If your robots.txt is generated (a Next.js robots.ts, say), the new group is a few lines. Watch the order of Allow and Disallow if you carve out exceptions.
• If you build crawlers, honoring this token is likely to become table stakes.

What to watch

The shared AI-training signal, from announcement to enforcement

1. Oct. 30, 2026Google, Microsoft, OpenAI and Anthropic back the signal; IETF draft -00 filed (Anthropic already honors it)
2. Dec. 15, 2026Bing Webmaster Tools transition settings open
3. Feb. 1, 2027Microsoft begins honoring the signal
4. Mar. 1, 2027Bing stops reading NoArchive/NoCache as AI preferences
5. May 3, 2027Current draft expires — watch for a revision or working-group adoption

• OpenAI's promised implementation details, and Google's progress updates

Closing

Building apps alone, I also keep a landing page and this blog running, and robots.txt has always been a file I write once and forget. Reading this, it struck me that the file has sat there for three years quietly defaulting to yes.

The line takes a minute to add. Before spending that minute, it's worth deciding what you actually want: no training only, or no AI citations either.

Which would you pick for your site?

Sources

• IETF Internet-Draft: draft-kelley-aipref-training-preference-signal-00 (Oct 30, 2026) — https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/
• Google Search Central Blog, "Supporting a shared AI training control for websites" (Oct 30, 2026) — https://developers.google.com/search/blog/2026/10/ai-training-control
• Microsoft Bing Blog, "Your content, your choice: clearer preferences for training and grounding" (Oct 30, 2026) — https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/
• OpenAI, "Helping websites express AI training preferences" (Oct 30, 2026) — https://openai.com/index/helping-websites-express-ai-training-preferences
• Kevin Kelley, post to the IETF ai-control list — https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w/
• Engadget, "Google, OpenAI, Anthropic and Microsoft back a shared AI crawling opt-out" (Oct 31, 2026) — https://www.engadget.com/2303225/google-openai-anthropic-ai-crawling-controls/
• ZDNet Korea, coverage of the announcement (Nov 1, 2026) — https://zdnet.co.kr/view/?no=20261101134510
• Unite.AI, "OpenAI to Honor Shared Robots.txt Signal for AI Training Preferences" (Oct 30, 2026) — https://www.unite.ai/openai-to-honor-shared-robots-txt-signal-for-ai-training-preferences/