A single shared line can soon keep your site out of AI training at Google, OpenAI, Microsoft and Anthropic. Here is what to paste, and what it won't stop.
Google, OpenAI, Microsoft and Anthropic have promised to honor one shared robots.txt rule that tells AI companies not to train on a website. Site owners add a group under the User-Agent name __usage_ai-training with a Disallow line, instead of writing rules for each company's crawler.
The catch is timing. Anthropic is building support now, OpenAI says the coming months, Microsoft starts on February 1, 2027 and Google aims for early 2027. Until then, keep company rules such as GPTBot, ClaudeBot and Google-Extended in place.
Microsoft's older NoArchive and NoCache tags stop counting for AI use after March 1, 2027, so Bing-focused sites should switch before then.
One short group of lines in your robots.txt file will soon tell Google, OpenAI, Microsoft and Anthropic, all at once, that your website is off limits for AI training. If you want to block AI crawlers with robots.txt right now, though, you still need a separate rule for each company. The smart move is to add both: the new shared group and the old per-crawler lines.
That matters if you run a blog, a shop or a small news site. Until now, saying no to AI training meant hunting down every company’s crawler name and writing rules for each one. The new signal is a “policy token” called __usage_ai-training. It comes from a draft that Anthropic’s standards lead, Kevin Kelley, filed on October 30 with the Internet Engineering Task Force (IETF), the group that writes many of the internet’s technical rules. The catch: none of the big three support it yet, and the dates differ by company.
You block AI crawlers with robots.txt by adding rule groups to a plain text file that sits in the top-level folder of your website, the same file search engines already read. Here is the order we’d follow.
The shared group looks like any other robots.txt group. The only new part is the name on the User-Agent line. To keep your whole site out of AI training, add:
User-Agent: __usage_ai-training
Disallow: /
You can carve out exceptions with a normal Allow line underneath, pointing at a folder you are happy for AI models to learn from, such as a press page. The draft adds no new commands and keeps the same path matching rules that robots.txt has always used (they are written down in RFC 9309), so older tools simply skip a group they don’t understand.
Then add the company groups you want, each as its own block with a Disallow: / line:
One warning from Anthropic: blocking its IP addresses instead of using robots.txt may not work reliably, because the bot then can’t read your preferences.
The shared token is a statement of how you want your content used, not an order about who may visit. The draft says so plainly. That has a few practical effects people miss:
It also says nothing about money. Microsoft says outright that its new preferences do not set up any licensing or payment arrangement.
No, blocking AI training does not remove you from regular Google or Bing results, according to both companies. Google’s documentation says Google-Extended has no effect on whether a site is included in Google Search. When a site uses the shared token and doesn’t mention Google-Extended, Google says it will treat that as if the site had disallowed Google-Extended.
There is a trade-off, though. Google says sites that specifically allow Google-Extended can appear in Gemini app answers and earn clicks from them. Block it, and that door closes. AI Overviews and AI Mode in Google Search are handled by a different switch, a generative AI control inside Search Console.
Microsoft splits things further. Next to the training preference, it is adding a page-level grounding preference with three options: Allow, Disallow and a new DisallowExtended. That last one keeps your pages usable in Microsoft’s own products, such as Copilot and Bing’s AI answers, while shutting out the grounding services Microsoft sells to other businesses. Microsoft warns that either restriction will cut how often your brand turns up in AI answers. Your spot in classic Bing results stays the same.
Each company has its own timetable, so adding the line today is about being ready, not instantly protected.
Microsoft’s switch has a deadline attached. The NoArchive and NoCache tags it has used for AI control since 2023 stop counting for training and grounding after March 1, 2027. Sites still relying on those tags that want to stay out of grounding can record a temporary choice in Bing Webmaster Tools, which opens on December 15. Once Microsoft spots either new preference on a site, it ignores the old tags there.
One open question remains for OpenAI users. Neither OpenAI’s announcement nor its crawler page explains what happens if a site’s shared group and its GPTBot group disagree. Keep them consistent and you avoid the problem.
For most small sites, the decision comes down to one question: do you want AI tools to learn from your writing, or to quote and link to it? Blocking training is now cheap, and it won’t cost you regular search traffic. Blocking grounding is the step that can make you disappear from AI answers, so think before you flip that one.
The new rule also doesn’t cover everyone. Meta’s communications head, Andy Stone, voiced support for a common signal but didn’t say Meta would honor it. Other AI companies aren’t part of the promise at all, which is why some site owners lean on network-level tools: Cloudflare already blocks AI crawlers by default, and the Really Simple Licensing (RSL) standard lets publishers charge bots for access. Cloudflare’s chief executive has warned that bot traffic could reach 1,000 times human traffic, so the pressure on small sites is only growing.
If you use a hosted website builder, check its help pages for how it lets you edit robots.txt before you assume you can paste these lines in.
It’s a shared policy token that lets a site state one AI training preference to every company that honors it. You write it on a User-Agent line and use Allow and Disallow as normal.
No. ChatGPT search uses OAI-SearchBot, a separate setting. You can block GPTBot for training and still allow OAI-SearchBot.
No. Google says Google-Extended doesn’t affect inclusion in Google Search, though it can keep you out of Gemini app answers.
You're reading the quick version.
Read next
Xbox Game Pass November 2026: every date, and the cloud-hour catch
Seven games are lined up between November 2 and 11, none of them for Xbox One, while new monthly limits on cloud streaming kick in.
Mumbled dialogue, voices that land late and a soundbar that goes silent all have a fix hiding in the same Fire TV menu.
The toggle, four ways in, two ways out, the fix when the setting isn't there, and the new controller shortcuts Microsoft is testing now.
Signed-in devices are only one way into your inbox. Delegates, forwarding rules and linked apps can read your mail too, and most people never look.
The Daily Brief
Every morning, the news that matters from AI, phones, apps and the people shaping tech. Explained in plain English. Free.
One email a day. No spam, unsubscribe anytime. Privacy policy
