On October 30, Google Search Central and Microsoft Bing both announced support for a shared robots.txt syntax for saying "don't use my content to train AI". It comes from an Internet-Draft that Anthropic's K. Kelley submitted the same day to the IETF's AI Preferences (AIPREF) working group, and its core is a new "policy token", __usage_ai-training: a site writes it once and every participating AI provider honours it, with no need for separate rules for each company's crawler. According to Search Engine Journal, OpenAI also says it will honour it; SEJ notes that this is still an individual draft rather than official IETF standardisation, and that none of the three companies supports it yet.

The Syntax and the Rules

  • It is an ordinary robots.txt group with no new directives. Opting the whole site out of training is User-Agent: __usage_ai-training followed by Disallow: /; the draft's example adds Allow: /press-kit/, meaning only that folder may be used for training.
  • A wildcard group such as User-Agent: * does not count as a training preference, and a file without a __usage_ai-training group expresses no preference at all.
  • It expresses how content may be used and is not a crawling directive: to limit who crawls and how fast, keep using each crawler's own group. Older parsers that don't recognise the token treat it as an unknown crawler name and ignore it, so existing rules work as before.

How Google and Bing Will Apply It

  • Google: when a site uses the token to opt out of training and does not specify Google-Extended separately, Google plans to treat it as if Google-Extended were disallowed; existing Google-Extended rules keep working. Google notes that sites that allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps, so opting out with the token also means giving up grounding in Gemini Apps on Google's side, a point SEJ also makes. Google aims to support it, along with other AI providers, in 2027.
  • Bing: honoured as pages are crawled from February 1, 2027. Besides the robots.txt training preference there is a page-level form, <meta name="Bingbot" content="ai-training:disallow"> or the header X-Robots-Tag: Bingbot: ai-training:disallow, plus a separate AI grounding preference with Allow, DisallowExtended and Disallow options. None of these changes a site's eligibility for regular Bing search results.
  • Bing's new preferences replace NoArchive and NoCache: sites using those tags should switch by March 1, 2027, after which they are no longer read as training or grounding preferences, and from December 15, 2026 such sites can also record a transition setting in Bing Webmaster Tools.

What Site Owners Can Do Now

Decide what you want first. To stay in search results but opt out of training, add the group above and leave your Googlebot and Bingbot crawl rules alone; to be cited in AI answers, don't block Google-Extended and leave Bing's grounding preference at its default of Allow. After editing robots.txt, remember Bing's caveat that a setting only takes effect once it recrawls the page; notifying it through IndexNow improves the chance of a recrawl but guarantees no date. Background reading: Cloudflare's trust scores for crawlers in this piece, its default blocking of AI crawlers here, zero-click AI search in this piece, and the bandwidth cost of AI crawlers here.