The AI Opt-Out Standard vs Real robots.txt Files

Home › Guides › The AI Opt-Out Standard vs Real robots.txt Files

✓ checked 8 October 2026 · sources: our fetch of robots.txt from Chrome’s 10,000 most-visited origins (CrUX, August 2026), IETF Datatracker, Cloudflare announcements · method · caveats · download the data

The IETF is writing a standard way for websites to tell AI systems whether their content may be used for training. Cloudflare changed how it writes robots.txt for its customers on 15 September. We fetched robots.txt from the 10,000 most-visited origins in Chrome’s user data to see which methods sites actually use today.

Read this first. The IETF drafts aren’t finished. The vocabulary draft says of its core definitions: “This section does not yet have consensus”. Low use of a draft syntax is expected at this stage, and robots.txt lines are requests, not technical blocks.

robots.txt files checked 6,735using the IETF draft’s Content-Usage rule 13blocking GPTBot by name 950

Headline numbers

1. 13 of 6,735 robots.txt files (0.19%) use the IETF draft’s Content-Usage rule. 12 of them are Vinted’s country sites.
2. 122 (1.8%) carry Cloudflare’s Content-Signal line, and 108 of those set ai-train=no.
3. The older method is far more common: 950 (14.1%) fully disallow OpenAI’s GPTBot by name, about 73 times as many as use Content-Usage. Among the top 1,000 origins it’s 129 of 730 (17.7%).
4. The IETF vocabulary draft reached revision -08 on 14 Sep 2026, its 11th version counting the individual draft it grew from. The working group’s 2 milestones, sending the vocabulary and the attachment specs to the IESG, were both due 31 Aug 2026 and are still listed as open.
5. 49 origins still serve the old “Cloudflare Managed content” block three weeks after Cloudflare said Managed robots.txt would be replaced by Bot Preference Sync. We found 0 with the “Cloudflare Bot Preference Sync” marker shown in Cloudflare’s example.

Which methods sites use

Where the IETF standard stands

The AI Preferences (aipref) working group has two documents headed for Proposed Standard. The vocabulary draft defines three categories of use, with short labels: AI Training (train-ai), AI Use (ai-use) and Search (search). The attachment draft says how to publish them, either as an HTTP header or as a robots.txt rule. Its example robots.txt is:

User-Agent: *
Allow: /
Content-Usage: train-ai=n

The vocabulary draft is at -08 (14 Sep 2026) and the attachment draft at -05 (19 Aug 2026). Datatracker lists 9 drafts in the group in total, including individual proposals. Neither working-group draft has been sent to the IESG for approval yet.

What Cloudflare changed

On 15 September 2026 Cloudflare introduced a “Disallow AI Training” setting. It says the setting “is named for the Disallow: directive it publishes in your robots.txt”, and that “Apple, Google, and Microsoft honor or have committed (in a specified time frame) to honor this setting.” It also said “Managed Robots.txt will be deprecated in favor of Bot Preference Sync.” Cloudflare says “17% of sites choose to enable some mechanism to block training” among its customers.

Cloudflare also wrote that it isn’t adding a Disallow setting for agents yet: “As standards such as ai-prefs mature, we will revisit this approach.” Its earlier Content-Signal line, introduced in September 2025, uses its own labels (search, ai-input, ai-train), which differ from the IETF draft’s.

For an earlier, larger survey of how robots.txt files treat named AI crawlers, see our AI-Crawler & llms.txt Census. For the court cases that test what scrapers may do, see how scraping cases end.

Method

Sites. All 10,000 origins in the top three Chrome UX Report popularity buckets (top 1,000, 5,000 and 10,000) for August 2026, from the public CrUX dataset on BigQuery. Origins aren’t sites: one company can have many (Vinted has a dozen country domains).

Fetching. On 8 October 2026 we requested /robots.txt from each origin once, following redirects, with a user agent identifying ProxyPicker. We counted a file as valid if it returned HTTP 200, wasn’t HTML and contained at least one robots.txt directive: 6,735 of 10,000 did.

Counting. A file counts for Content-Usage or Content-Signal if it contains at least one such line. A crawler is “fully disallowed” if a group naming it has Disallow: / and no Allow: /. Cloudflare blocks were detected by their comment markers.

Verification. A separate script re-reads the saved responses with an independent parser and recounts every number.

Caveats

• We fetched once, from one location. Some sites block unknown clients or vary robots.txt by visitor; about a third of origins returned no valid file.
• robots.txt only expresses a preference; crawlers decide whether to follow it.
• Sites can also signal AI preferences in HTTP headers, page metadata or terms of service, which we didn’t check.
• The IETF drafts may still change, including their labels.
• Cloudflare’s example marker was “shortened and anonymized”, so its real marker may read differently from what we searched for.
• Cloudflare’s figures are the company’s own and weren’t independently checked.

Download the data

• robots.txt AI preferences (CSV): all 10,000 origins with fetch status, any Content-Usage and Content-Signal lines, the Cloudflare marker and four named-crawler blocks.
• IETF aipref draft revisions (CSV): every revision of the two working-group drafts and their predecessors.

Cite as: ProxyPicker, “The AI Opt-Out Standard vs Real robots.txt Files”, 8 October 2026, https://proxypicker.com/robots-txt-ai-preferences/. Origin list: Chrome UX Report (Google), CC BY 4.0. Draft data: IETF Datatracker.

Related: AI-Crawler & llms.txt Census · LinkedIn v. ProAPIs and how scraping cases end · reCAPTCHA vs Turnstile · Wikipedia’s Bot Traffic · Web Bot Auth: Standards Tracker (the IETF effort on bot identity) · Common Crawl Robots.txt Denials, 2016–2026.

Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.