Home › Guides › Health Insurers Must Let Web Crawlers Reach Price Files
✓ checked 28 October 2026 · sources: Federal Register (final rule 2026-20447), one request to each of 12 insurer pages · method · caveats · download the data
A US federal rule now says health plans’ price files must be open to “any person, automated script, or web crawler”, with no CAPTCHAs, 403 errors or download limits in the way. The Transparency in Coverage final rule from the Treasury, Labor and Health and Human Services departments was published in the Federal Register on 6 October 2026 (91 FR 63748). It takes effect on 7 December 2026 and applies to plan years beginning on or after 1 January 2027.
“web crawler” in the rule: 7×Insurer pages that loaded: 8 of 12robots.txt allows the page: 10 of 10
Why this matters
Since 2022, US health plans and insurers have had to post large machine-readable files listing the prices they have agreed with doctors and hospitals. The files are often many gigabytes, and they are mostly read by software, not people. The new rule is one of the few places where US federal regulation says outright that a website must not block automated access. It names common anti-bot tools as barriers.
What the rule says
The new regulation text says the files “must be publicly available and accessible to any person, automated script, or web crawler free of charge and without conditions, such as establishment of a user account, password, submission of personally identifiable information or other credentials, or blocking server configurations or firewalls to access the file.” The departments proposed the wording in plural form and switched to the singular in the final rule “for grammatical clarity”.
In the preamble, the departments list examples of barriers: asking a user to enter an EIN, TIN or plan name to reach a file, making them encounter a “CAPTCHA” or “a 403 error”, and “limits on the number of downloads allowed by a user or at a time”.
What commenters said
- Several commenters supported the change, citing “non-crawlable URLs and download throttling”. A few named CAPTCHAs as an ongoing obstacle.
- One commenter said some plans and insurers block “requests from IP address ranges from major cloud providers”. Another asked the departments to say that “publicly available” means “accessible via straightforward HTTP requests”.
- One commenter opposed the requirement on security grounds and asked that plans be allowed “reasonable technical safeguards”.
The departments kept the “without conditions” wording. They wrote that “access to the machine-readable files comes predominantly from automated systems” and that “CAPTCHAs and rate limiting” are barriers to automated ingestion. They added that in enforcement they will consider “whether a plan was trying to protect itself from distributed denial of service attacks”.
Spot check: 12 insurer pages
On 28 October we sent one request to the price-file page of each of 12 large insurers, using a user agent that names ProxyPicker, and read each site’s robots.txt. We did not try to get past anything. 8 pages loaded. The rule does not apply yet, and these are the pages that link to the files, not the files themselves.
| Insurer | HTTP status | Result | robots.txt for this page |
|---|---|---|---|
| UnitedHealthcare | 200 | page loaded | allowed |
| Aetna | TLS error | certificate could not be verified | not checked |
| Cigna | 200 | page loaded | allowed |
| Anthem | 200 | page loaded | allowed |
| Humana | 200 | page loaded | allowed |
| Blue Cross and Blue Shield of Illinois | 200 | page loaded | allowed |
| Kaiser Permanente | 302 | redirected to a regional page | allowed |
| Centene | 200 | page loaded | allowed |
| Blue Cross Blue Shield of Massachusetts | 404 | not found at this address | allowed |
| Highmark | 404 | not found at this address | allowed |
| Molina Healthcare | 200 | page loaded | no file |
| Oscar Health | 200 | page loaded | allowed |
All 10 robots.txt files we could read allow the page. Molina’s robots.txt address returned a web page, not a robots file, and we could not verify Aetna’s security certificate, so we did not check its robots.txt.
For US law on scraping, see the scraping law hub. Government sites’ own use of CAPTCHAs is covered in federal websites’ CAPTCHA vendors, and anti-bot vendors in the anti-bot census.
Method
We read the rule’s full text and details from the Federal Register API on 28 October 2026 and counted the phrase “web crawler”. For the spot check, we sent one GET request to each page with curl, with a 25-second timeout and without following redirects, and fetched each host’s robots.txt once. We checked the page path against the robots.txt rules for all user agents, using longest-match rules.
Caveats
- The rule applies from plan years beginning on or after 1 January 2027. Until then the earlier rules apply.
- A page loading for one request says nothing about rate limits, the file hosts or the files.
- Two pages returned 404 at the addresses we had; we did not search for new addresses. The Kaiser Permanente page redirects to a regional page, which we did not follow.
- This is a description of the rule, not legal advice.
Download the data
Cite as: ProxyPicker, “Health Insurers Must Let Web Crawlers Reach Price Files”, 28 October 2026, https://proxypicker.com/health-price-files-crawler-rule/.
Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.