Home › Guides › Which Federal .gov Sites Block AI Bots
✓ checked 18 October 2026 · sources: CISA’s .gov domain list (federal domains, last changed 16 October 2026), one robots.txt request per domain on 18 October · method · caveats · download the data
Only 22 of the 449 federal .gov domains with a readable robots.txt file block an AI bot by name. We checked every domain in CISA’s list of 1,321 federal .gov domains for 18 AI user agents. GPTBot, OpenAI’s training crawler, is the most blocked, on 15 domains. Congress.gov blocks 17 of the 18, in a single group that lists 125 different user agents.
22 of 449 block an AI botGPTBot blocked on 15user-triggered agents blocked on 9
Headline numbers
- 4.9% of readable federal robots.txt files block at least one AI bot by name (22 domains run by 17 organizations). 25 name one at all.
- Training crawlers are blocked more than agents. 22 domains block at least one training crawler; 9 block a user-triggered agent, and ChatGPT-User is blocked on 9 of them. OpenAI’s search crawler, OAI-SearchBot, is blocked on 3.
- Congress.gov and sciencebase.gov block 17 of the 18 AI user agents we checked, including ChatGPT-User, Claude-User and Perplexity-User. 7 domains block just one.
- 24 domains block all crawlers with a
User-agent: *rule, which covers AI bots without naming them.
The 22 domains
| Domain | Organization (CISA list) | AI agents blocked (of 18) | Types |
|---|---|---|---|
| congress.gov | Library of Congress | 17 | 9 training, 3 AI search, 5 user-triggered |
| sciencebase.gov | Department of the Interior | 17 | 10 training, 2 AI search, 5 user-triggered |
| uscongress.gov | Library of Congress | 17 | 9 training, 3 AI search, 5 user-triggered |
| america.gov | Department of State | 15 | 10 training, 5 user-triggered |
| reportband.gov | Department of the Interior | 15 | 10 training, 5 user-triggered |
| americabydesign.gov | Executive Office of the President | 10 | 10 training |
| cftc.gov | Commodity Futures Trading Commission | 10 | 10 training |
| fgdc.gov | Department of the Interior | 10 | 10 training |
| techprosperitycorps.gov | Peace Corps | 10 | 10 training |
| oversight.gov | Council of the Inspectors General on Integrity and Efficiency | 8 | 8 training |
| usagm.gov | United States Agency for Global Media | 5 | 4 training, 1 user-triggered |
| ahcpr.gov | Department of Health and Human Services | 4 | 3 training, 1 user-triggered |
| ahrq.gov | Department of Health and Human Services | 4 | 3 training, 1 user-triggered |
| arts.gov | National Endowment for the Arts | 2 | 1 training, 1 user-triggered |
| gao.gov | Government Accountability Office | 2 | 1 training, 1 AI search |
| asc.gov | Appraisal Subcommittee of the Federal Financial Institutions Examination Council | 1 | 1 training |
| citap.gov | Department of Energy | 1 | 1 training |
| dfc.gov | United States International Development Finance Corporation | 1 | 1 training |
| energystar.gov | Environmental Protection Agency | 1 | 1 training |
| ncpc.gov | National Capital Planning Commission | 1 | 1 training |
| va.gov | Department of Veterans Affairs | 1 | 1 training |
| vets.gov | Department of Veterans Affairs | 1 | 1 training |
3 pairs share one file through redirects: ahcpr.gov and ahrq.gov, va.gov and vets.gov, and congress.gov and uscongress.gov. VA’s file blocks only Amazonbot, with the comment “block Amazon’s AI training crawler”. Going the other way, ipp.gov’s file explicitly allows GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
Context
The New York Times reported on 25 September 2026 that OpenAI’s AI systems had “meddled with the websites for the Education Department, the Commerce Department and the Securities and Exchange Commission” that summer without the company’s knowledge, according to security researchers and a person familiar with the episodes. The paper said OpenAI confirmed the Commerce Department and SEC incidents. Those episodes were described as involving AI agents rather than crawlers. robots.txt is a voluntary signal that doesn’t technically stop access, but it is the standard way for a site to ask particular bots to stay away.
For the same check across the wider web, see our AI crawler census and the AI opt-out standard vs real robots.txt files. To check whether a visit really came from an AI company, see AI fetcher IP ranges.
Method
We took the 1,321 domains in CISA’s current-federal.csv and requested https://domain/robots.txt, then https://www.domain/robots.txt if that failed, once each on 18 October 2026, with a user agent that identifies ProxyPicker. 449 returned a readable robots.txt file (416 distinct files after redirects). The rest gave no response (336), a 404 (148), a 403 (102), another error, or a 200 page that wasn’t a robots.txt file (271, usually an HTML page). We parsed the files following RFC 9309: a bot is “blocked” when the group naming it disallows / with no Allow rule. A bot listed together with * in a group that only blocks some paths is counted as named, not blocked. The 18 user agents are training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, cohere-ai), AI search crawlers (PerplexityBot, OAI-SearchBot, Claude-SearchBot) and user-triggered agents (ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, meta-externalfetcher).
Caveats
- A domain is not a website: many federal domains redirect elsewhere or serve nothing, and big agencies run many subdomains we didn’t check.
- Some servers block unfamiliar clients, so a few 403s and timeouts may be caused by our request rather than the site.
- robots.txt shows what a site asks for. It doesn’t show whether bots comply, or what firewalls block.
- One day’s snapshot. Files change without notice.
Download the data
- Every federal .gov domain (CSV): robots.txt status, AI bots named, blocked and allowed.
- By user agent (CSV).
Cite as: ProxyPicker, “Which Federal .gov Sites Block AI Bots”, 18 October 2026, https://proxypicker.com/federal-gov-ai-bots-robots-txt/. Domain list: CISA dotgov-data (CC0).
Related: AI crawler census · AI opt-out standard vs real robots.txt · AI fetcher IP ranges.
Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.