Home › Guides › What the Newspapers Suing OpenAI Block in robots.txt
✓ checked 23 October 2026 · sources: the live robots.txt files of the 19 publications named in the complaint, the complaint in USA TODAY Co. v. OpenAI (S.D.N.Y. 1:26-cv-08892), the ai.robots.txt community list · method · caveats · download the data
Every one of the 19 newspapers in USA TODAY Co.’s copyright suit against OpenAI tells 276 named bots to stay off the entire site, and the list is identical on all of them. OpenAI’s GPTBot, ChatGPT-User and OAI-SearchBot are on it, as are Anthropic’s ClaudeBot and Google-Extended. But OpenAI’s agents, along with Google-Extended and PerplexityBot, get a short list of exceptions: sponsored, branded, shopping and contributor sections they are still allowed to read.
publications checked: 19user agents fully disallowed on every site: 276scraping and proxy tokens among them: 90
Disallow: / rule on all 19 sites, sorted into a type by ProxyPicker. Fetched 23 October 2026.Why this matters now
USA TODAY Co., the publisher formerly known as Gannett, and 13 of its subsidiaries sued seven OpenAI entities in the Southern District of New York on 8 October 2026, alleging copyright infringement over content from 19 publications. The complaint says those publications “also block OpenAI’s web crawlers using robots.txt”. Copyright suits against AI companies keep arriving: on 21 October, Billboard reported two new suits brought by the music publisher Reservoir, one against Suno and one against Anthropic. Our AI copyright lawsuits tracker follows the cases. This page checks what the robots.txt files of the 19 publications actually say today.
What the files say
- One shared file. 279 user agents are named on 18 of the sites. The Oklahoman names 289, because it also lists Googlebot, Bingbot and other search and social crawlers with partial rules. The same 276 agents get
Disallow: /everywhere, and across all 19 files 289 distinct user agents are named. - OpenAI by name. 10 of the blocked tokens refer to OpenAI or ChatGPT, including GPTBot, ChatGPT-User, OAI-SearchBot, ChatGPT Agent and ChatGPT Operator.
- Exceptions for sponsored content. 6 blocked agents also have
Allowlines: GPTBot, ChatGPT-User, OAI-SearchBot, ChatGPT Operator, Google-Extended and PerplexityBot. Across the 19 sites there are 18 distinct allowed paths. Every site allows at least 4, including branded-story and contributor-content sections. USA TODAY itself allows 10, among them its shopping and money-management sections. Several sites also allow specific sponsor-story articles one by one. - Scraping and proxy companies. 90 of the blocked tokens belong to 39 scraping APIs, proxy networks and scraping frameworks, including Bright Data, Oxylabs, Decodo (formerly Smartproxy), SOAX, Zyte, ScrapingBee, ScraperAPI, ZenRows, Apify and Scrapy. A robots.txt rule only works if the client checks it and sends an honest user agent. Our publisher lobbying on AI crawlers page covers why publishers are pushing for rules with more force.
- Search engines and archives are blocked too. 20 search-engine tokens, including Applebot, Baiduspider, YandexBot and Seznam, plus 4 Internet Archive and Arquivo.pt tokens, are fully disallowed. Googlebot and Bingbot are not.
- Coverage of the community list. The ai.robots.txt project’s list named 192 distinct user agents on 23 October. 106 of them are fully blocked on all 19 sites, and 85 are not named at all. Most of the missing ones are newer agents and coding tools.
Why imgproxy is not counted as a scraper
One token on the list, imgproxy, looks at first like a proxy or scraping brand. It isn’t one. imgproxy is open-source image-resizing server software. A site runs it on its own infrastructure, and it fetches source images to resize them. According to its configuration documentation, the default User-Agent it sends is imgproxy/ followed by its version number. It is not a crawler and does not discover pages, so we put it in its own category and leave it out of the scraping and proxy count.
Method
On 23 October 2026 we fetched https://www.<domain>/robots.txt for the 19 publications named in paragraph 2 of the complaint. All returned HTTP 200 with a plain-text file. We grouped consecutive User-agent lines into rule groups, lower-cased each token and recorded its Disallow and Allow lines. An agent counts as fully disallowed when its group contains Disallow: /. Each fully disallowed token was given one type by hand, and the CSV records that type, so you can regroup them. A token counts as being on the ai.robots.txt list when it matches, ignoring case, a User-agent line in that project’s robots.txt on the same day. Six scraping-tool tokens appear on both our scraping list and the ai.robots.txt list, and the chart counts them only as scraping. For a similar check on other sites, see our retail robots.txt and AI agents study.
Caveats
- robots.txt is a request, not an access control. These files say nothing about whether any company complied, and nothing on this page alleges that anyone ignored them.
- We could not get archived copies of these files, so this page does not say when any rule was added.
- Assigning a token to a company is based on its name and public documentation. Some tokens are generic, such as
spider, and are marked as unidentified. - The complaint’s allegations have not been tested in court.
Download the data
- Per publication: counts, key agents and Allow exceptions (CSV).
- Every named user agent with its type and rules (CSV).
Cite as: ProxyPicker, “What the Newspapers Suing OpenAI Block in robots.txt”, 23 October 2026, https://proxypicker.com/usatoday-openai-lawsuit-robots-txt/.
Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.