Home › Anti-Bot Census
✓ data: HTTP Archive crawl of 1 September 2026 (15,470,478 mobile home pages), plus 11 earlier crawls back to January 2022 · analysed 2026-10-02 · method · download the data
How much of the web screens visitors for bots, and with what? We counted the bot-defense signals that show up on 15,470,478 website home pages in the public HTTP Archive crawl: bot-detection cookies set by Cloudflare, Akamai, DataDome, HUMAN (PerimeterX), Kasada, Imperva and AWS WAF, CAPTCHA scripts such as reCAPTCHA and Turnstile, and the response headers of bot-filtering firewalls. Then we split the results by site popularity and went back to 2022.
One thing we learned on the way: the easy way to measure this gets Cloudflare badly wrong. More than half the Cloudflare bot cookies found on home pages are set by someone else’s embedded widget, not by the site. We count only the ones a site sets for itself.
- home pages analysed 15,470,478
- any bot-defense signal 22.5%
- top-10k sites 35.1%
- enterprise anti-bot, top 10k 7.7%
- Cloudflare bot cookie since 2024 2.8×
The headline numbers
- About 1 in 4.5 websites screens visitors for bots on its home page. 22.5% of home pages (3,476,370 of 15,470,478) show at least one bot-defense signal: a bot-detection cookie set by the site itself, a CAPTCHA script, or a bot-filtering firewall header.
- Among the 10,000 most-visited sites it’s more than 1 in 3. 35.1% of top-10k home pages (2,630 of 7,500) show a signal, against 22.3% for sites outside the top million.
- Paid enterprise anti-bot tools are about 20× more common at the top. First-party cookies from Akamai Bot Manager, DataDome, HUMAN, Kasada, Imperva or AWS WAF appear on 7.7% of top-10k home pages (574 of 7,500), but on only 0.38% beyond the top million. At the top, Akamai leads (3.8%).
- Cloudflare bot screening on home pages has nearly tripled in two years. The share of home pages that set Cloudflare’s
__cf_bmbot cookie themselves went from 2.7% in July 2024 (432,652 pages) to 7.6% in September 2026 (1,168,725 pages). - Counting every
__cf_bmcookie would more than double the Cloudflare number. 16.4% of home pages end up holding one, but on 54% of those pages (1,365,915 of 2,534,640) it comes only from a third-party embed, such as the LinkedIn or X tracking tags, whose own servers sit behind Cloudflare. In a hand check of 15 random top-100k pages flagged this way, 11 got the cookie only from another domain. - reCAPTCHA still rules CAPTCHAs, but Turnstile is catching up. Google’s reCAPTCHA is on 89% of the 2,359,820 home pages with a CAPTCHA script. Its share of all home pages slipped from 16.4% (January 2023) to 13.6%, while Cloudflare Turnstile grew from 0.01% to 1.21%. In July 2024 there were 146 reCAPTCHA sites for every Turnstile site; now there are 11.
By site popularity
HTTP Archive groups sites by their Chrome UX Report popularity rank, in buckets (top 1,000, top 5,000 and so on). Each row below is one bucket, so “1k–5k” means ranks 1,001 to 5,000. Some sites in each bucket weren’t crawled successfully, which is why the counts are below the bucket sizes.
| Popularity bucket | Home pages | Any signal | CAPTCHA script | Cloudflare bot cookie (first-party) | Enterprise anti-bot | Akamai | DataDome | HUMAN | Imperva | AWS WAF |
|---|---|---|---|---|---|---|---|---|---|---|
| Top 1k | 754 | 35.0% | 20.2% | 10.2% | 8.0% | 2.9% | 1.1% | 0.4% | 0.4% | 3.1% |
| 1k–5k | 3,006 | 35.3% | 20.7% | 9.8% | 8.0% | 4.1% | 1.4% | 0.5% | 0.7% | 1.3% |
| 5k–10k | 3,740 | 34.9% | 22.2% | 8.5% | 7.3% | 3.6% | 1.1% | 0.4% | 1.1% | 1.2% |
| 10k–50k | 29,679 | 32.9% | 21.5% | 9.1% | 5.0% | 2.1% | 0.6% | 0.3% | 1.2% | 0.8% |
| 50k–100k | 37,629 | 31.1% | 21.2% | 8.9% | 3.7% | 1.2% | 0.4% | 0.2% | 1.3% | 0.5% |
| 100k–500k | 309,944 | 26.8% | 19.1% | 8.1% | 1.9% | 0.5% | 0.2% | 0.05% | 0.9% | 0.4% |
| 500k–1M | 398,098 | 24.4% | 17.4% | 7.6% | 1.3% | 0.2% | 0.07% | 0.02% | 0.7% | 0.3% |
| 1M–5M | 3,330,370 | 22.9% | 16.4% | 7.1% | 0.6% | 0.09% | 0.04% | 0.01% | 0.3% | 0.1% |
| 5M–10M | 4,315,834 | 22.3% | 15.6% | 7.2% | 0.3% | 0.04% | 0.05% | 0.02% | 0.2% | 0.08% |
| 10M+ | 7,041,424 | 22.0% | 14.2% | 7.9% | 0.3% | 0.02% | 0.03% | 0.04% | 0.1% | 0.08% |
| Top 10k combined | 7,500 | 35.1% | 21.4% | 9.2% | 7.7% | 3.8% | 1.2% | 0.4% | 0.9% | 1.4% |
| All sites | 15,470,478 | 22.5% | 15.3% | 7.6% | 0.5% | 0.06% | 0.04% | 0.03% | 0.2% | 0.1% |
Bot walls since 2022
__cf_bm cookie, including ones set by embedded widgets). The gold line counts only cookies set by the site itself; that needs HTTP Archive’s cookie data, which we pulled for three crawls.The two 2022 crawls covered fewer sites (7.9 and 10.5 million home pages, against 15–16 million since 2023), so the jump between them partly reflects a different sample. Compare 2023 onwards with each other.
| Crawl | Home pages | reCAPTCHA | Turnstile | hCaptcha | Any __cf_bm cookie (naive) | Cloudflare bot cookie, first-party | Enterprise anti-bot, first-party |
|---|---|---|---|---|---|---|---|
| 2022-01 | 7,919,113 | 11.1% | 0.00% | 0.02% | 4.8% | – | – |
| 2022-07 | 10,537,208 | 17.8% | 0.00% | 0.10% | 5.6% | – | – |
| 2023-01 | 16,029,741 | 16.4% | 0.01% | 0.13% | 5.7% | – | – |
| 2023-07 | 16,331,285 | 15.7% | 0.03% | 0.17% | 7.2% | – | – |
| 2024-01 | 15,060,956 | 15.8% | 0.06% | 0.20% | 7.4% | – | – |
| 2024-07 | 16,022,393 | 15.9% | 0.11% | 0.23% | 7.8% | 2.7% | 0.37% |
| 2025-01 | 15,033,500 | 16.1% | 0.20% | 0.29% | 10.0% | – | – |
| 2025-07 | 15,425,851 | 16.2% | 0.31% | 0.35% | 12.0% | 4.0% | 0.42% |
| 2026-01 | 14,744,764 | 15.0% | 0.51% | 0.42% | 11.3% | – | – |
| 2026-03 | 15,854,786 | 14.9% | 0.60% | 0.43% | 11.5% | – | – |
| 2026-06 | 15,706,731 | 14.4% | 0.73% | 0.42% | 15.1% | – | – |
| 2026-09 | 15,470,478 | 13.6% | 1.21% | 0.50% | 16.3% | 7.6% | 0.46% |
Two things stand out. Cloudflare’s first-party bot cookie gained the most ground, 4.9 percentage points since July 2024 (Turnstile grew faster in relative terms, but from a tiny base). And CAPTCHAs are holding flat or shrinking: the share of home pages with any CAPTCHA script was 16.5% in January 2023 and 15.3% in September 2026. One reading: sites are moving bot checks out of the page and into the network, where visitors never see a puzzle.
The CAPTCHA market
We only see CAPTCHAs that load on the home page. Many sites put them on login, sign-up or checkout pages instead, which HTTP Archive doesn’t crawl, so the real totals are higher. The split between brands is probably the more reliable figure.
What a scraper will run into
Each row is one technology, with the signal we used to spot it and where it’s most common. “Top 10k” means the 10,000 most-visited sites.
| Technology | What we detected | All home pages | Share of all | Share of top 10k |
|---|---|---|---|---|
| Cloudflare bot products (Bot Fight Mode, Super Bot Fight Mode, Bot Management) | __cf_bm cookie set for the site’s own domain | 1,168,725 | 7.55% | 9.2% |
| Akamai Bot Manager | _abck, bm_sz, ak_bmsc or bm_sv cookie (first-party) | 9,424 | 0.06% | 3.8% |
| Imperva (Incapsula) | incap_ses_*, visid_incap_*, nlbi_* or reese84 cookie (first-party) | 33,412 | 0.22% | 0.9% |
| AWS WAF bot control / challenge | aws-waf-token cookie (first-party) | 16,630 | 0.11% | 1.4% |
| DataDome | datadome cookie (first-party) | 6,831 | 0.04% | 1.2% |
| HUMAN (PerimeterX) | _px3, _pxhd, _pxvid or related cookie (first-party) | 4,364 | 0.03% | 0.4% |
| Kasada | KP_UIDz cookie (first-party) | 121 | 0.00% | 0.1% |
| DDoS-Guard | __ddg* cookie (first-party) | 83,706 | 0.54% | 0.6% |
| reCAPTCHA | reCAPTCHA script or widget on the page | 2,110,052 | 13.64% | 19.4% |
| Cloudflare Turnstile | Turnstile JavaScript object on the page | 187,713 | 1.21% | 1.6% |
| hCaptcha | hCaptcha script or widget on the page | 77,509 | 0.50% | 0.6% |
| Sucuri firewall | x-sucuri-id response header | 34,780 | 0.22% | 0.1% |
Top-10k column: 7,500 crawled home pages. For the smaller vendors that means a handful of sites (for example 8 for Kasada), so read those shares as rough.
Why most Cloudflare bot counts are too high
Cloudflare says it sets the __cf_bm cookie on visitors to sites “protected by Bot Management or Bot Fight Mode” (Cloudflare docs). The open-source Wappalyzer rules that HTTP Archive uses report “Cloudflare Bot Management” whenever that cookie shows up on a page, for any domain.
That matters because many popular embeds are themselves served through Cloudflare with bot screening on. Load a news site with a LinkedIn Insight tag or an X (Twitter) pixel and your browser picks up a __cf_bm cookie for linkedin.com or twitter.com, even if the news site has never heard of Cloudflare. Across the September 2026 crawl, 2,534,640 home pages held a __cf_bm cookie, but only 1,168,725 had one for their own domain. The naive count reports 16.3%; the first-party count is 7.6%.
We also checked by hand. Out of 15 randomly picked top-100k home pages that Wappalyzer flags as using Cloudflare Bot Management, only 4 set the cookie for their own domain. The other 11 got it only from other domains: mostly LinkedIn, X and OneSignal tags, and in one case the site’s own image server on a separate domain.
Sidebar: do scraping companies screen bots?
We ran the same first-party check on the home pages of 17 proxy and scraping-API companies (September 2026 crawl, mobile and desktop). Bright Data and ScraperAPI set Cloudflare’s __cf_bm bot cookie on their own domain. ProxyEmpire, Scrape.do and Proxy-Cheap set Cloudflare’s cf_clearance cookie, which Cloudflare uses for its challenge and JavaScript bot checks. For 11 of the 17, every bot cookie we found came from a third-party embed, mostly HubSpot, LinkedIn, X, G2 or Capterra widgets. None showed first-party cookies from Akamai, DataDome, HUMAN, Kasada or Imperva.
This only looks at home pages. Sign-up, login and dashboard pages often run tighter checks; our AI-Crawler Census sidebar found several vendors using CAPTCHAs on exactly those pages.
Methodology
- Data: HTTP Archive’s monthly crawl, table
httparchive.crawl.pagesin Google BigQuery. We used home pages (is_root_page) from the mobile crawl: 15,470,478 for 1 September 2026, and 11 earlier crawls from January 2022 to June 2026 for the trend. HTTP Archive loads each page once in Chrome from US data centres, without logging in or clicking anything. - CAPTCHA and firewall signals come from HTTP Archive’s technology detections (Wappalyzer rules): script URLs, JavaScript objects, page markup and response headers. reCAPTCHA, Turnstile, hCaptcha, Friendly Captcha, GeeTest, Yandex SmartCaptcha, MTCaptcha and five smaller CAPTCHA services count as “CAPTCHA script”. Imperva, Sucuri, DDoS-Guard, Variti and AntiBot.Cloud headers count as firewall signals.
- Bot-detection cookies come from HTTP Archive’s cookie data (
custom_metrics.cookies). A cookie counts only if its domain belongs to the same registrable domain as the page (so.shop.example.comcounts forwww.example.com, but.linkedin.comdoesn’t). The 2023 crawls don’t carry this cookie data, so first-party figures start in July 2024. We pulled three crawls: July 2024, July 2025 and September 2026. - “Enterprise anti-bot” means a first-party cookie from Akamai Bot Manager, DataDome, HUMAN (PerimeterX), Kasada, Imperva or AWS WAF. These are paid products. Cloudflare is counted separately because its cookie also appears with the free Bot Fight Mode.
- “Any bot-defense signal” means any first-party bot cookie above (including Cloudflare and DDoS-Guard), any CAPTCHA script, or any of the firewall headers.
- Desktop check: the September 2026 desktop crawl (11,359,325 home pages) gives similar shares: CAPTCHA script 16.4% on desktop against 15.3% on mobile, Turnstile 1.13% against 1.21%.
- Popularity: Chrome UX Report rank buckets, as recorded by HTTP Archive for each crawl.
- Cost and reproducibility: 21 BigQuery queries, 134 GB billed in total. Every query and its raw output are saved, and the CSVs below are built directly from them.
Limitations
- A signal isn’t a product tier. A Cloudflare cookie can mean the free Bot Fight Mode or the enterprise Bot Management add-on; we can’t tell which. A CAPTCHA script on the page doesn’t mean every visitor is challenged.
- We undercount. Defenses that leave no cookie, script or header on the home page (network-level rules, server-side fingerprinting, checks only on login or checkout pages) are invisible here. Sites can also rename or disable vendor cookies.
- Some sites block the crawler itself. When a bot wall stops HTTP Archive’s browser, the page it records may be a challenge page. That still shows a signal, but the rest of that page’s data is missing.
- Detection rules change. Wappalyzer rules are updated over time, and older crawls were analysed with the rules of their day. Vendor-level trends before 2024 (for example DataDome, which shows almost no detections before 2024) partly reflect rule changes, so we don’t quote them.
- Spot check. We re-fetched 140 sampled pages once with a plain HTTP client (no JavaScript) on 2 October 2026. Header- and cookie-based signals were often visible again (Imperva 15 of 15); script-based ones mostly weren’t (reCAPTCHA 3 of 15), which is expected because they load through JavaScript. The results are in the spot-check CSV.
- We don’t use Cloudflare Radar figures. Radar measures traffic, not site configuration, and its data licence doesn’t fit our use.
Download the data
- By popularity bucket, September 2026 (CSV): page counts and shares for every signal.
- Trend, January 2022 to September 2026 (CSV)
- Every Wappalyzer “Security” technology detected, September 2026 (CSV)
- Spot-check sample (CSV)
Free to use with attribution (CC BY 4.0). Please cite the crawl date.
Cite or embed
Suggested citation: ProxyPicker (2026). “Anti-Bot Census: bot-defense signals on 15,470,478 website home pages.” HTTP Archive crawl of 1 September 2026, analysed 2 October 2026. https://proxypicker.com/anti-bot-census/
Embed the chart:
<a href="https://proxypicker.com/anti-bot-census/"><img src="https://proxypicker.com/wp-content/uploads/2026/10/proxypicker-antibot-by-rank-2026-10.png" alt="Share of websites with bot-defense signals by popularity (ProxyPicker Anti-Bot Census)" width="800" /></a><br />Source: <a href="https://proxypicker.com/anti-bot-census/">ProxyPicker Anti-Bot Census</a> (HTTP Archive, 1 September 2026)Data: HTTP Archive (httparchive.org), Chrome UX Report. Detection rules: HTTP Archive’s fork of Wappalyzer.
Related: Anti-Bot vs Proxy Patents (who patents bot detection, fingerprinting and CAPTCHA) · AI-Crawler & llms.txt Census (who blocks AI bots in robots.txt) · Best proxies for web scraping · Best scraping APIs · Best residential proxies · The Price of a Gigabyte · The Scraping Workforce (who does scraping work and what they earn) · Proxy & Scraping Search Demand Report (what people search for, 2022–2026).
Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.