Home › Guides › Common Crawl Robots.txt Denials
✓ checked 12 October 2026 · source: Common Crawl crawl statistics (Apache-2.0) · method · caveats · download the data
Robots.txt files blocked 6.16% of Common Crawl’s fetch attempts in its 9 crawls of 2026, up from 3.39% in 2023 and 1.43% in 2018. In the September 2026 crawl (CC-MAIN-2026-39) the crawler was turned away by robots.txt 193.2 million times out of 3.49 billion attempts (5.53%).
robots-denied, Sep 2026 crawl 193.2 million2026 vs 2023 share 1.8×crawls covered 104
Headline numbers
- The share has nearly doubled since 2023. Pooled over each year’s crawls it went from 3.39% in 2023 to 6.16% in 2026, 1.8 times as high. The median crawl tells the same story: 2.70% in 2023 against 5.74% in 2026.
- The typical 2026 crawl logged 197.5 million robots denials (median), against 113.1 million in 2021, while fetch attempts per crawl have stayed between 3.27 billion and 4.65 billion since 2019.
- One spike: CC-MAIN-2026-21 had 338.0 million robots denials (9.49%), the highest share of any crawl since 2017. The next crawl fell back to about 6%. The statistics don’t say why.
Robots.txt denials by year
Context
These counters come from CCBot, Common Crawl’s crawler. On 15 September 2026 Cloudflare said in a press release that it is replacing its “Managed Robots.txt” feature with “Bot Preference Sync”, which writes robots.txt rules from a site’s AI settings. The September crawl ran from 4 to 17 September, according to Common Crawl’s index, so it can’t show any effect of that change.
For how individual sites word their AI rules, see the AI opt-out survey and the AI-Crawler Census. For automated traffic on Wikipedia, see Wikipedia’s Bot Traffic. The same statistics show how often the crawler connects over IPv6: see IPv6 in Common Crawl.
Crawls since 2024
| Crawl | Fetch attempts | Robots-denied | Share |
|---|---|---|---|
| CC-MAIN-2026-39 | 3,491,290,715 | 193,203,472 | 5.53% |
| CC-MAIN-2026-34 | 3,446,735,235 | 197,059,037 | 5.72% |
| CC-MAIN-2026-30 | 3,454,759,774 | 205,080,000 | 5.94% |
| CC-MAIN-2026-25 | 3,406,561,317 | 202,073,665 | 5.93% |
| CC-MAIN-2026-21 | 3,562,105,566 | 337,993,269 | 9.49% |
| CC-MAIN-2026-17 | 3,585,327,457 | 217,210,179 | 6.06% |
| CC-MAIN-2026-12 | 3,282,647,598 | 188,473,300 | 5.74% |
| CC-MAIN-2026-08 | 3,483,275,559 | 197,522,208 | 5.67% |
| CC-MAIN-2026-04 | 3,616,559,010 | 192,151,953 | 5.31% |
| CC-MAIN-2025-51 | 3,409,457,525 | 196,266,083 | 5.76% |
| CC-MAIN-2025-47 | 3,585,452,894 | 219,573,638 | 6.12% |
| CC-MAIN-2025-43 | 3,720,822,711 | 191,204,170 | 5.14% |
| CC-MAIN-2025-38 | 3,481,594,608 | 187,537,000 | 5.39% |
| CC-MAIN-2025-33 | 3,579,195,444 | 175,894,025 | 4.91% |
| CC-MAIN-2025-30 | 3,619,679,510 | 181,114,878 | 5.00% |
| CC-MAIN-2025-26 | 3,592,271,667 | 159,227,497 | 4.43% |
| CC-MAIN-2025-21 | 3,674,018,218 | 152,743,792 | 4.16% |
| CC-MAIN-2025-18 | 4,047,863,325 | 184,069,115 | 4.55% |
| CC-MAIN-2025-13 | 4,020,443,907 | 169,830,613 | 4.22% |
| CC-MAIN-2025-08 | 3,887,374,247 | 186,368,376 | 4.79% |
| CC-MAIN-2025-05 | 4,226,325,712 | 176,545,258 | 4.18% |
| CC-MAIN-2024-51 | 3,740,993,208 | 170,882,396 | 4.57% |
| CC-MAIN-2024-46 | 3,738,908,543 | 151,620,445 | 4.06% |
| CC-MAIN-2024-42 | 3,523,794,847 | 155,472,674 | 4.41% |
| CC-MAIN-2024-38 | 3,906,633,226 | 158,606,948 | 4.06% |
| CC-MAIN-2024-33 | 3,290,059,251 | 146,820,848 | 4.46% |
| CC-MAIN-2024-30 | 3,579,486,425 | 152,654,706 | 4.26% |
| CC-MAIN-2024-26 | 3,986,216,346 | 157,638,112 | 3.95% |
| CC-MAIN-2024-22 | 3,746,403,860 | 145,180,193 | 3.88% |
| CC-MAIN-2024-18 | 3,848,217,167 | 150,646,294 | 3.91% |
| CC-MAIN-2024-10 | 4,326,140,264 | 165,875,440 | 3.83% |
Method
Common Crawl publishes crawler counters for every crawl in its cc-crawl-statistics repository on GitHub. On 12 October 2026 we downloaded all 104 files (CC-MAIN-2016-18 to CC-MAIN-2026-39). For each crawl we divide the robots-denied counter by the total fetcher counter. The counter is called fetcher:robots_denied up to CC-MAIN-2026-08 and fetcher:robots_denied_total from CC-MAIN-2026-12; the share didn’t jump at the change. Yearly figures pool all crawls in the year. A separate script recomputes every number from the raw files.
Caveats
- These are counts of URL fetches, not of sites. One large site that blocks CCBot can account for many denials, and the crawler’s choice of URLs changes from crawl to crawl.
- Crawls before 2017 used a different setup and show much higher shares (up to 12.8% in 2016), so the chart starts in 2017.
- The figures describe one crawler. Other crawlers are blocked at different rates.
- “Access denied” responses (HTTP 403 and similar, 1,257,762 in the September crawl) are separate and not included.
Download the data
- Common Crawl robots denials, 2016–2026 (CSV): every crawl with fetch attempts, successes, robots denials, share and access denials.
Cite as: ProxyPicker, “Common Crawl Robots.txt Denials, 2016–2026”, 12 October 2026, https://proxypicker.com/common-crawl-robots-denials/. Data: Common Crawl crawl statistics.
Related: The AI Opt-Out Standard vs Real robots.txt Files · AI-Crawler & llms.txt Census · Wikipedia’s Bot Traffic · IPv6 in Common Crawl.
Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.