Do Europe’s Privacy Regulators Block AI Crawlers?

Home › Guides › Do Europe’s Privacy Regulators Block AI Crawlers?

✓ checked 27 October 2026 · sources: robots.txt files of 33 data-protection regulators, HTTP Archive September 2026 crawl, npm and PyPI download statistics · method · caveats · download the data

Europe’s data-protection regulators are about to finish consulting on guidance that tells AI companies to respect robots.txt, but only 1 of the 24 regulators whose robots.txt we could read uses its own file to address an AI crawler. That one is the Dutch Autoriteit Persoonsgegevens, which blocks OpenAI’s GPTBot from its whole site. The European Data Protection Board itself sets rules only for all bots in general (User-agent: *).

Regulators with readable robots.txt: 24 of 33EU-domain sites with a GPTBot rule: 4.6%.com sites with a GPTBot rule: 10.3%

Why this matters now

The European Data Protection Board’s draft Guidelines 03/2026 on web scraping in the context of generative AI are open for comment until 23:59 CET on 30 October 2026. The draft says companies collecting training data should exclude “websites which clearly oppose the scraping of their content” through measures that include “robots.txt or ai.txt files, or CAPTCHA”. It also says that the “non-applicability of a robots.txt file on a web site does not amount to consent within the meaning of the GDPR”. In other words, a site that does not block crawlers has not agreed to have personal data scraped from it.

What the regulators’ own robots.txt files say

  • One AI rule. The Dutch regulator’s file contains the comment “prevent AI from using the site for learning”, followed by User-agent: GPTBot and Disallow: /. No other regulator names GPTBot, ClaudeBot, CCBot, Google-Extended, Meta’s crawlers or any other AI crawler.
  • Rules for search-marketing bots instead. Luxembourg’s CNPD blocks eight named crawlers: trendictionbot, SplitSignalBot and six SemrushBot variants. The UK ICO blocks the deepcrawl site-audit crawler, and Italy’s Garante has groups for Googlebot, Bingbot, Baiduspider and Twitterbot.
  • No AI preference signals. None of the readable files uses Cloudflare’s Content-Signal line, and none blocks all bots from the whole site.
  • 9 gave us no file to read. Finland, Lithuania and Croatia showed an automated bot check page instead of the file. Portugal, Slovenia and Iceland returned “not found”. Norway returned a server error, Cyprus redirected to an HTML page on gov.cy, and Bulgaria did not respond. Under the robots.txt standard (RFC 9309), a missing file means crawlers may fetch everything.

How the wider web compares

HTTP Archive’s September 2026 crawl recorded the robots.txt of 15.5 million site home pages. Of 2,845,274 sites on EU country domains (the 27 member-state domains plus .eu), 131,965 (4.6%) had a Disallow rule for GPTBot. That compares with 666,684 of 6,444,797 .com sites (10.3%) and 8.9% of sites on all other domains.

Share of root pages whose robots.txt has a Disallow rule for GPTBot, HTTP Archive September 2026.com top 10k.com top 10k: 18.8%18.8%EU top 10kEU top 10k: 32.1%32.1%.com, all.com, all: 10.3%10.3%Other TLDs, allOther TLDs, all: 8.9%8.9%EU, allEU, all: 4.6%4.6%
Share of home pages whose robots.txt has at least one Disallow line in a group for GPTBot. “Top 10k” is the most popular rank bucket in HTTP Archive’s data, which comes from the Chrome UX Report. Source: HTTP Archive, crawl of 1 September 2026 (mobile).
  • The pattern flips among popular sites. Among the most-visited sites, EU domains were more likely to have a GPTBot rule: 316 of 985 (32.1%), against 645 of 3,422 .com sites (18.8%).
  • Same gap for other crawlers. Meta-ExternalAgent rules appeared on 4.0% of EU-domain sites and 8.4% of .com sites. For ClaudeBot the figures were 4.2% and 9.7%, and for Common Crawl’s CCBot 4.2% and 9.8%.

The code that reads these files

A robots.txt rule only works if the crawler’s software checks it. The most-downloaded open-source parsers show how routine that check has become:

  • robots-parser (npm, MIT licence) was downloaded 19,635,841 times in September 2026, 2.8 times its October 2025 total of 7,014,239. In the 30 days to 26 October it had 20,971,124. robots-txt-guard had 407,114 in September.
  • Protego (PyPI), the parser used by the Scrapy crawling framework, had 7,021,321 downloads in September and 10,340,433 in the 30 days to 26 October, excluding mirrors. robotspy had 600,201 and the older reppy 1,922 in September.

All 33 regulators

RegulatorDomainrobots.txt resultNamed user agents
European Data Protection Boardedpb.europa.eureadableonly User-agent: *
European Data Protection Supervisorwww.edps.europa.eureadableonly User-agent: *
Austriawww.dsb.gv.atreadableonly User-agent: *
Belgiumwww.autoriteprotectiondonnees.bereadableonly User-agent: *
Bulgariawww.cpdp.bgno response–
Croatiaazop.hrbot check page (200)–
Cypruswww.dataprotection.gov.cyredirects to an HTML page on gov.cy–
Czechiauoou.gov.czreadableonly User-agent: *
Denmarkwww.datatilsynet.dkreadableonly User-agent: *
Estoniawww.aki.eereadableonly User-agent: *
Finlandtietosuoja.fibot check page (403)–
Francewww.cnil.frreadableonly User-agent: *
Germany (federal)www.bfdi.bund.dereadableonly User-agent: *
Greecewww.dpa.grreadableonly User-agent: *
Hungarynaih.hureadableonly User-agent: *
Irelandwww.dataprotection.iereadableonly User-agent: *
Italywww.garanteprivacy.itreadableGooglebot, Bingbot, Baiduspider, Twitterbot
Latviawww.dvi.gov.lvreadableonly User-agent: *
Lithuaniavdai.lrv.ltbot check page (403)–
Luxembourgcnpd.public.lureadabletrendictionbot, SemrushBot-SA, SemrushBot-BA, SemrushBot-SI, SemrushBot-SWA, SemrushBot-CT, SemrushBot-BM, SplitSignalBot
Maltaidpc.org.mtreadableonly User-agent: *
Netherlandswww.autoriteitpersoonsgegevens.nlreadableGPTBot
Polanduodo.gov.plreadableonly User-agent: *
Portugalwww.cnpd.ptnot found (404)–
Romaniawww.dataprotection.roreadableonly User-agent: *
Slovakiadataprotection.gov.skreadableonly User-agent: *
Sloveniawww.ip-rs.sinot found (404)–
Spainwww.aepd.esreadableonly User-agent: *
Swedenwww.imy.sereadableonly User-agent: *
Norwaywww.datatilsynet.noserver error (500)–
Icelandwww.personuvernd.isnot found (404)–
Liechtensteinwww.datenschutzstelle.lireadableonly User-agent: *
United Kingdomico.org.ukreadabledeepcrawl

For US government sites, see which federal websites block AI bots. How AI preference signals beyond robots.txt are spreading is covered in robots.txt AI preferences, and our sample of Common Crawl’s robots.txt files is in Common Crawl robots.txt denials.

Method

On 27 October 2026 we requested /robots.txt from the main website of the EDPB, the EDPS, the data-protection authority of each EU member state (for Germany, the federal BfDI), and the authorities of Norway, Iceland, Liechtenstein and the UK. We followed redirects and used an ordinary browser user agent. A file counted as readable if it returned HTTP 200 with plain-text robots.txt rules. We listed every user-agent group and compared names against a list of 17 known AI crawler tokens.

HTTP Archive figures are totals from its public BigQuery table (httparchive.crawl.pages, crawl date 1 September 2026, mobile client, root pages only), using the parsed robots.txt data that HTTP Archive stores for each page. A site counts for a crawler if its robots.txt has at least one Disallow line in a group naming that crawler, even if the rule covers only part of the site. Sites are grouped by the domain’s top-level domain, not by where the owner is based. Data from HTTP Archive. Download counts come from the npm registry downloads API and from pypistats.org, excluding mirrors.

Caveats

  • A regulator’s robots.txt reflects how its website is managed. It is not a statement of legal policy.
  • Robots.txt files change. Croatia’s file was readable in an earlier check on the morning of 27 October but showed a bot check page later that day.
  • Bot check pages can block crawlers without any robots.txt rule, and we did not test them further.
  • Download counts include automated installs and build pipelines, so they are not counts of crawlers.

Download the data

Cite as: ProxyPicker, “Do Europe’s Privacy Regulators Block AI Crawlers?”, 27 October 2026, https://proxypicker.com/eu-privacy-regulators-robots-txt/. HTTP Archive data: HTTP Archive, September 2026 crawl.

Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.