For years, a site that wanted to say no to robots.txt AI training had to chase every crawler by name. GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and so on. Miss one and your content was fair game.

That may finally change. On October 30, OpenAI, Google and Microsoft said they will honor one shared robots.txt line for AI training preferences. Anthropic, whose engineer wrote the proposal, has already adopted it.

I am Nitin from the Debugged Pro team. We manage websites for clients in India and the USA, so this one lands on our desk directly. Below are the verified facts, the exact syntax, the dates, and what we plan to do for client sites.

Robots.txt AI training opt-out example using the __usage_ai-training policy token with Disallow and Allow rules — The shared AI training signal, using the example from the IETF draft. Graphic: Debugged Pro original.

What Happened: One Line Instead of Ten

As The Verge reported on the new AI scraping opt-out standard, the biggest AI companies have agreed to read a common signal. Publishers can now set their training preference once, instead of writing separate rules for each provider.

The idea comes from Kevin Kelley, a technical lead for standards at Anthropic. He published it as an Internet-Draft with the Internet Engineering Task Force (IETF), the group that writes many of the web's core standards.

OpenAI announced its support in a short post titled Making it easier for websites to express AI training preferences. It says the signal is voluntary and "will not replace the controls we already provide through robots.txt."

Here is who has committed to what, based on The Verge, Search Engine Journal and the companies' own posts:

Anthropic: has already adopted the signal.
OpenAI: aims to implement it "in the coming months." GPTBot rules still apply.
Google: aims to support it in 2027. Its crawler docs say early 2027.
Microsoft: will honor the new preferences from February 1, 2027.
Meta: supports the idea but has not said it will honor it.

Meta communications head Andy Stone told The Verge: "We believe creating a common signal that conveys such preferences to multiple providers at once is an important step." That is support in words, not a firm promise.

How the Robots.txt AI Training Signal Works

The clever part is that nothing new is added to robots.txt itself. There are no new directives and no new grammar. Instead, the draft invents a special name to put in the User-Agent line.

That name, called a policy token, is __usage_ai-training. You then write normal Allow and Disallow rules under it. Here is the example from the draft:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

This says: please do not train on anything on this site, except the press kit folder. The usual path rules and longest-match logic from the Robots Exclusion Protocol standard, RFC 9309 still apply.

Because old parsers ignore groups for bots they do not know, this does not break anything that reads robots.txt today.

What counts as AI training

The draft uses the definition from the IETF AI Preferences working group. AI training means using content to change the learned parameters of a generative AI model. Models that only do non-generative jobs, like classification or ranking, are outside that definition.

Four rules that will trip people up

We read the IETF draft on the machine-readable training preference signal closely. These are the points most likely to cause mistakes on client sites:

It is not a crawl block. The draft says plainly that the token "is not a directive for crawling behavior." Crawlers may still visit your pages.
Wildcards do not count. A User-agent: * group must not be read as matching the token. An old blanket block says nothing about training here.
Silence means no preference. If no group names the token, the site has stated no preference at all, neither yes nor no.
Partial blocks can backfire. If you only disallow one folder, you are stating a preference in favor of training on every other path. Block the root first, then list exceptions.

Private agreements between a site and an AI company can also override the signal, according to The Next Web's reading of the draft. So a licensing deal still wins.

Training, Grounding and Search Are Three Different Things

This is where most confusion will come from. AI companies use your pages in three separate ways:

Training: your content changes the model itself.
Grounding: a model reads your page live while writing an answer.
Search: your page is indexed and shown in results.

The new line covers training only. That matters a lot for SEO. A client can say no to training and still appear in search and in many AI answers.

Table comparing how OpenAI, Google, Microsoft, Anthropic and Meta will handle the robots.txt AI training signal and their timelines — Who has committed to the shared signal and when, based on The Verge, Search Engine Journal and company posts. Graphic: Debugged Pro original.

Company	Support for the shared signal	What else to know
Anthropic	Already adopted	Its engineer wrote the draft
OpenAI	In the coming months	GPTBot handles training, OAI-SearchBot handles ChatGPT search
Google	2027	Token opt-out treated as a Google-Extended block if Google-Extended is not set
Microsoft	From February 1, 2027	Adds a separate page-level grounding setting
Meta	Supportive, no commitment	No date given

Google's twist

Google says that if a site opts out with the token and does not mention Google-Extended, it will treat that as a Google-Extended block. Search Engine Journal notes this also covers grounding in Gemini Apps. Sites that explicitly allow Google-Extended can still appear in Gemini answers and get clicks.

Google's AI Overviews and AI Mode follow normal Googlebot rules. So a training opt-out alone should not remove a site from them.

Microsoft's twist

Microsoft is adding a second setting for AI grounding, set at the page level. A meta tag or HTTP header can also set the training preference for one page and override robots.txt.

From February 1, 2027, Bing will honor the new options as it crawls.
After March 1, 2027, it will stop reading the old NoArchive and NoCache tags as training or grounding choices.
Sites stay in Bing search results whatever they choose.

Microsoft also gave a clear warning. Blocking AI grounding "will significantly reduce your content's visibility in AI-generated experiences." It added that the new preferences "do not establish licensing or payment arrangements."

What the Robots.txt AI Training Change Means for Agencies in India and the USA

Most of our clients are not news publishers. They are SaaS firms, D2C brands, clinics and B2B service companies. For them, being quoted in AI answers can bring real leads. Blocking everything by reflex could cost them traffic.

Other clients feel the opposite. A course creator or a research firm may not want its paid work feeding a model. Both choices are fair. What is not fair is an agency making the choice by accident.

We saw the same pattern with shopping bots. As we wrote in our guide on why retailers are blocking AI shopping agents, an audit found many big US stores block agents through old bot rules nobody revisited. Robots.txt files tend to get written once and forgotten.

Indian agencies that run sites for US brands should note the Microsoft deadlines in particular. A NoArchive tag a team added in 2023 for AI reasons will stop working for that purpose after March 1, 2027.

A Practical Plan for Client Sites

Here is the checklist we are using with clients this quarter:

Audit the current file. List every AI crawler rule in robots.txt, plus any NoArchive or NoCache tags and CDN bot settings.
Split the decision in three. Ask the client separately about training, grounding and search. Write each answer down.
Keep existing vendor rules. Do not delete GPTBot or Google-Extended lines. OpenAI and Google say those keep working.
Add the token group carefully. If the client opts out, block the root first, then allow any folders they want used, like a press kit.
Plan for Bing. Put the February and March 2027 dates in the client calendar and move off the old tags in time.
Test before you ship. Run the new file through a robots.txt tester and check that search crawlers are still allowed.
Review every six months. The draft is still changing, and so are company policies.

Seven step checklist for agencies to set robots.txt AI training preferences on client websites — A rollout checklist for agency teams in India and the USA. Graphic: Debugged Pro original.

A sample file for a client that opts out

This is a simple starting point. It keeps search open, blocks named training bots as before, and adds the new shared line:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

User-agent: *
Allow: /

Adjust it to the client's real choices. If they want to be cited in Gemini answers, for example, do not block Google-Extended.

The Caveats You Should Tell Clients

Be honest about the limits. This is not a law and not a lock on the door.

The draft is an individual submission. The IETF says it is not endorsed and has no formal standing yet. It expires on May 3, 2027 unless updated.
It only works with companies that choose to honor it. Smaller scrapers may ignore it.
It does not undo training that already happened.
OpenAI, Google and Microsoft do not support it yet, based on their own stated timelines.

For clients with strict data rules, robots.txt should be one layer among several. Our responsible AI governance team can help set a wider policy for content and data use.

Our Take

This is a good, practical step. One line is far easier to manage than a growing list of crawler names. It also forces a healthy conversation: what does each client actually want AI companies to do with their content?

Our advice is simple. Do not panic and do not block everything. Make the training, grounding and search choices on purpose, write them down, and update robots.txt once the client agrees.

Key Takeaways

OpenAI, Google and Microsoft will honor one shared robots.txt AI training signal, and Anthropic already does.
The signal uses the __usage_ai-training token with normal Allow and Disallow rules.
It covers training only, not crawling, grounding or search.
A User-agent: * block does not count, and a one-folder block can imply yes to the rest.
Microsoft starts honoring it on February 1, 2027 and drops old tags for AI use after March 1, 2027.

Want help reviewing robots.txt and AI crawler settings across your sites? Talk to our software development consulting team. We work with businesses across India and the USA.

About the Author

Nitin Kumar is the Lead Project Coordinator at Debugged Pro. He plans delivery for software, web and AI projects across India and the USA, and helps clients make clear, safe choices about new technology.