How do I opt out of AI training in robots.txt?

As of November 5, 2026, use two layers. First, add the shared policy token defined in the October 30, 2026 IETF Internet-Draft: a group with 'User-Agent: __usage_ai-training' and 'Disallow: /'. Google, Microsoft, OpenAI and Anthropic said on October 30 they will honor it, but Microsoft's effective date is February 1, 2027 and Google says 2027. Second, keep the per-company rules that work now: Disallow GPTBot (OpenAI training), ClaudeBot (Anthropic training) and Google-Extended (Gemini training), while allowing their search crawlers so you stay visible in AI answers.

Does the __usage_ai-training token stop AI crawlers from visiting my site?

No. The draft says the policy token is not a directive for crawling behaviour; it only expresses a preference about whether content may be used to train generative AI models. Crawlers still follow their own User-Agent groups for access. To stop a crawler from fetching pages, disallow its product token, and to enforce it, block it at your CDN, because robots.txt compliance is voluntary.

Will opting out of AI training remove my site from ChatGPT, Gemini or Copilot answers?

Not if you separate training from search. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search, so you can block one and allow the other. Anthropic splits ClaudeBot (training) from Claude-SearchBot and Claude-User. Microsoft is adding a separate page-level AI Grounding preference, so training and grounding are independent. The exception is Google: it says only sites that allow Google-Extended are eligible to appear and receive clicks via grounding in Gemini Apps, so disallowing Google-Extended has a visibility cost there.

When do Google and Bing start honoring the shared AI training opt-out?

Microsoft Bing starts honoring the new AI Training and AI Grounding preferences on February 1, 2027 and asks publishers to act by March 1, 2027, after which NoArchive and NoCache no longer count as training or grounding preferences. Google's October 30, 2026 Search Central post says it aims to support the control in 2027 without a specific date; when it does, a training opt-out through the token with no Google-Extended rule will be read as a Google-Extended disallow.

Quick Answer

How to Opt Out of AI Training in robots.txt (2026 Guide)

Published: November 5, 2026

The short answer

Write both the new shared opt-out and the per-company rules. The shared token is the future; the company-specific user agents are what is honored as of November 5, 2026. A complete training opt-out that keeps you in AI search looks like this:

# Shared AI training preference (IETF draft, Oct 30, 2026)
User-Agent: __usage_ai-training
Disallow: /

# Per-company training crawlers (honored now)
User-Agent: GPTBot
Disallow: /

User-Agent: ClaudeBot
Disallow: /

User-Agent: Google-Extended
Disallow: /

# Search and answer crawlers stay allowed
User-Agent: OAI-SearchBot
Allow: /

User-Agent: Claude-SearchBot
Allow: /

User-Agent: *
Allow: /

Read the Google caveat below before you disallow Google-Extended.

What changed on October 30, 2026

An individual Internet-Draft, draft-kelley-aipref-training-preference-signal-00 (“Machine-Readable Training Preference Signal”), defines a robots.txt policy token, __usage_ai-training, written in the User-Agent line of an ordinary group. It adds no new directives and no new grammar: path matching, longest-match precedence and group structure stay as they are, so existing parsers simply ignore the group. The draft expires on May 3, 2027 and is work in progress, not an RFC.

Paths work as usual. This example from the draft opts out of training for everything except a press kit:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Two properties matter in practice:

It is about use, not access. The draft states the token “is not a directive for crawling behavior.” A crawler that sees a training opt-out may still fetch the page for search.
It is independent of crawler rules. A bot still checks its own User-Agent group for crawl permission. Opting out of training does not block GPTBot; you do that with a GPTBot group.

Who honors it, and when

Company	Commitment (Oct 30, 2026)	Effective	What works now
Microsoft (Bing, Copilot)	Honors the shared AI Training preference in robots.txt, plus a page-level override and a separate page-level AI Grounding preference	February 1, 2027; act by March 1, 2027	NoArchive / NoCache (being replaced)
Google	Token opt-out with no Google-Extended rule is read as a Google-Extended disallow	”In 2027”, no date	`Google-Extended`
OpenAI	Said it will honor the shared signal	No date published	`GPTBot` (training) vs `OAI-SearchBot` (search)
Anthropic	Said it will honor the shared signal	No date published	`ClaudeBot` (training) vs `Claude-SearchBot`, `Claude-User`

Microsoft’s grounding preference has two values: Disallow (no grounding use) and DisallowExtended, which limits grounding to Microsoft’s own experiences (Copilot, Microsoft 365) and excludes grounding services it sells to third parties. During the transition you also record a temporary grounding preference in Bing Webmaster Tools; it applies per page until recrawl or September 30, 2027, whichever comes first.

Step by step

Decide training and search separately. Most sites want to be cited in AI answers but not used for training. Those are different crawlers and, from 2027, different preferences.
Add the shared group (User-Agent: __usage_ai-training, Disallow: /). It costs nothing and becomes effective as each provider switches on support.
Add per-company training groups: GPTBot, ClaudeBot, Google-Extended. Per OpenAI’s docs, GPTBot and OAI-SearchBot settings are independent, and robots.txt changes take about 24 hours to reach ChatGPT search.
Allow the search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot (Perplexity says it is not used for training foundation models), Bingbot, Googlebot.
Weigh Google-Extended. Google says sites that allow Google-Extended are eligible to appear and receive clicks through grounding in Gemini Apps. Blocking it opts you out of Gemini training and that grounding. If Gemini visibility matters, leave Google-Extended allowed.
Know what robots.txt cannot do. User-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User) act for a person; OpenAI says robots.txt rules may not apply to ChatGPT-User. And compliance is voluntary. To enforce, block at the edge; see Akamai vs Cloudflare for AI bot scraping protection.
Before March 1, 2027: if you rely on NoArchive or NoCache with Bing, replace them with the new training and grounding preferences.

Common mistakes

Blocking * to stop AI. It removes you from every search engine. Name the AI crawlers instead.
Thinking the token blocks crawling. It does not; see step 3.
Blocking Claude-User or ChatGPT-User to stop training. Those fetch pages when a user asks; blocking them removes you from live answers without affecting training.
Forgetting subdomains. robots.txt is per host; Anthropic’s docs ask for the rule on every subdomain you want opted out.

If you use Cloudflare, its managed robots.txt can prepend AI directives for you; check the result at /robots.txt after saving. For shopping and checkout agents rather than crawlers, see how to control AI shopping agents on your website.

Last verified: November 5, 2026.

The short answer

What changed on October 30, 2026

Who honors it, and when

Step by step

Common mistakes

Sources