Which Federal .gov Sites Block AI Bots

Home › Guides › Which Federal .gov Sites Block AI Bots

✓ checked 18 October 2026 · sources: CISA’s .gov domain list (federal domains, last changed 16 October 2026), one robots.txt request per domain on 18 October · method · caveats · download the data

Only 22 of the 449 federal .gov domains with a readable robots.txt file block an AI bot by name. We checked every domain in CISA’s list of 1,321 federal .gov domains for 18 AI user agents. GPTBot, OpenAI’s training crawler, is the most blocked, on 15 domains. Congress.gov blocks 17 of the 18, in a single group that lists 125 different user agents.

22 of 449 block an AI botGPTBot blocked on 15user-triggered agents blocked on 9

Headline numbers

  1. 4.9% of readable federal robots.txt files block at least one AI bot by name (22 domains run by 17 organizations). 25 name one at all.
  2. Training crawlers are blocked more than agents. 22 domains block at least one training crawler; 9 block a user-triggered agent, and ChatGPT-User is blocked on 9 of them. OpenAI’s search crawler, OAI-SearchBot, is blocked on 3.
  3. Congress.gov and sciencebase.gov block 17 of the 18 AI user agents we checked, including ChatGPT-User, Claude-User and Perplexity-User. 7 domains block just one.
  4. 24 domains block all crawlers with a User-agent: * rule, which covers AI bots without naming them.
Federal .gov domains fully blocking each AI user agentGPTBotGPTBot: 1515CCBotCCBot: 1414ClaudeBotClaudeBot: 1313AmazonbotAmazonbot: 1212anthropic-aianthropic-ai: 1212BytespiderBytespider: 1111Google-ExtendedGoogle-Extended: 1111meta-externalagentmeta-externalagent: 1010ChatGPT-UserChatGPT-User: 99cohere-aicohere-ai: 99Applebot-ExtendedApplebot-Extended: 88Claude-UserClaude-User: 55meta-externalfetchermeta-externalfetcher: 55MistralAI-UserMistralAI-User: 55Perplexity-UserPerplexity-User: 55PerplexityBotPerplexityBot: 44OAI-SearchBotOAI-SearchBot: 33Claude-SearchBotClaude-SearchBot: 22
Federal .gov domains whose robots.txt fully disallows each AI user agent by name, 18 October 2026. Green: training crawlers; amber: AI search crawlers; pink: user-triggered agents. Source: CISA .gov data, robots.txt files.

The 22 domains

DomainOrganization (CISA list)AI agents blocked (of 18)Types
congress.govLibrary of Congress179 training, 3 AI search, 5 user-triggered
sciencebase.govDepartment of the Interior1710 training, 2 AI search, 5 user-triggered
uscongress.govLibrary of Congress179 training, 3 AI search, 5 user-triggered
america.govDepartment of State1510 training, 5 user-triggered
reportband.govDepartment of the Interior1510 training, 5 user-triggered
americabydesign.govExecutive Office of the President1010 training
cftc.govCommodity Futures Trading Commission1010 training
fgdc.govDepartment of the Interior1010 training
techprosperitycorps.govPeace Corps1010 training
oversight.govCouncil of the Inspectors General on Integrity and Efficiency88 training
usagm.govUnited States Agency for Global Media54 training, 1 user-triggered
ahcpr.govDepartment of Health and Human Services43 training, 1 user-triggered
ahrq.govDepartment of Health and Human Services43 training, 1 user-triggered
arts.govNational Endowment for the Arts21 training, 1 user-triggered
gao.govGovernment Accountability Office21 training, 1 AI search
asc.govAppraisal Subcommittee of the Federal Financial Institutions Examination Council11 training
citap.govDepartment of Energy11 training
dfc.govUnited States International Development Finance Corporation11 training
energystar.govEnvironmental Protection Agency11 training
ncpc.govNational Capital Planning Commission11 training
va.govDepartment of Veterans Affairs11 training
vets.govDepartment of Veterans Affairs11 training

3 pairs share one file through redirects: ahcpr.gov and ahrq.gov, va.gov and vets.gov, and congress.gov and uscongress.gov. VA’s file blocks only Amazonbot, with the comment “block Amazon’s AI training crawler”. Going the other way, ipp.gov’s file explicitly allows GPTBot, ClaudeBot, PerplexityBot and Google-Extended.

Context

The New York Times reported on 25 September 2026 that OpenAI’s AI systems had “meddled with the websites for the Education Department, the Commerce Department and the Securities and Exchange Commission” that summer without the company’s knowledge, according to security researchers and a person familiar with the episodes. The paper said OpenAI confirmed the Commerce Department and SEC incidents. Those episodes were described as involving AI agents rather than crawlers. robots.txt is a voluntary signal that doesn’t technically stop access, but it is the standard way for a site to ask particular bots to stay away.

For the same check across the wider web, see our AI crawler census and the AI opt-out standard vs real robots.txt files. To check whether a visit really came from an AI company, see AI fetcher IP ranges.

Method

We took the 1,321 domains in CISA’s current-federal.csv and requested https://domain/robots.txt, then https://www.domain/robots.txt if that failed, once each on 18 October 2026, with a user agent that identifies ProxyPicker. 449 returned a readable robots.txt file (416 distinct files after redirects). The rest gave no response (336), a 404 (148), a 403 (102), another error, or a 200 page that wasn’t a robots.txt file (271, usually an HTML page). We parsed the files following RFC 9309: a bot is “blocked” when the group naming it disallows / with no Allow rule. A bot listed together with * in a group that only blocks some paths is counted as named, not blocked. The 18 user agents are training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent, Amazonbot, anthropic-ai, cohere-ai), AI search crawlers (PerplexityBot, OAI-SearchBot, Claude-SearchBot) and user-triggered agents (ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, meta-externalfetcher).

Caveats

  • A domain is not a website: many federal domains redirect elsewhere or serve nothing, and big agencies run many subdomains we didn’t check.
  • Some servers block unfamiliar clients, so a few 403s and timeouts may be caused by our request rather than the site.
  • robots.txt shows what a site asks for. It doesn’t show whether bots comply, or what firewalls block.
  • One day’s snapshot. Files change without notice.

Download the data

Cite as: ProxyPicker, “Which Federal .gov Sites Block AI Bots”, 18 October 2026, https://proxypicker.com/federal-gov-ai-bots-robots-txt/. Domain list: CISA dotgov-data (CC0).

Related: AI crawler census · AI opt-out standard vs real robots.txt · AI fetcher IP ranges.

Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.