Which News Sites Block Meta’s Crawlers

Home › Guides › Which News Sites Block Meta’s Crawlers

✓ checked 24 October 2026 · sources: live robots.txt files of 65 English-language news sites, the ai.robots.txt community list, the News/Media Alliance letter of 22 October 2026 · method · caveats · download the data

Of 65 major news sites we checked, 38 tell Meta’s AI crawler, Meta-ExternalAgent, to stay off the whole site, about the same as the 34 that do so for OpenAI’s GPTBot and well below the 49 that block Anthropic’s ClaudeBot. None of them block facebookexternalhit, the fetcher Facebook uses to build link previews when an article is shared. That split matters this week: Meta’s revised Commercial Terms, due to take effect on 30 October, would let Meta crawl websites associated with business accounts, and publishers say robots.txt cannot opt them out of that.

news sites read: 65block Meta-ExternalAgent: 38block facebookexternalhit: 0

Of 65 news sites, how many disallow each crawler from the whole siteMeta-ExternalAgent (Meta)Meta-ExternalAgent (Meta): 3838FacebookBot (Meta)FacebookBot (Meta): 3535Meta-ExternalFetcher (Meta)Meta-ExternalFetcher (Meta): 2727Meta-WebIndexer (Meta)Meta-WebIndexer (Meta): 1414facebookexternalhit (Meta)facebookexternalhit (Meta): 00CCBotCCBot: 5050ClaudeBotClaudeBot: 4949Applebot-ExtendedApplebot-Extended: 4646PerplexityBotPerplexityBot: 4444Google-ExtendedGoogle-Extended: 3636GPTBotGPTBot: 3434Claude-UserClaude-User: 3232ChatGPT-UserChatGPT-User: 2525OAI-SearchBotOAI-SearchBot: 2020
Sites whose robots.txt applies Disallow: / to the crawler, either in a group naming it or, where it is not named, in the catch-all User-agent: * group. Includes files that add a few Allow exceptions. Meta crawlers in orange. Fetched 24 October 2026.

Why this matters now

On 15 October Nieman Lab reported that Meta’s new Commercial Terms authorise it to use “web scrapers, bots, robots, spiders, crawlers, and user agents” on domains associated with businesses that use Facebook, Instagram, WhatsApp or Messenger business tools. According to Nieman Lab, Meta removed the word “train” on 16 October and limited use to “advertising and business products”. On 22 October the News/Media Alliance, which says it represents more than 2,000 publishers, sent Mark Zuckerberg an open letter asking Meta to withdraw the terms before 30 October and to respond by 26 October. Meta’s position is set out in its terms; we did not contact Meta or the publishers. This page records what the publishers’ robots.txt files say today, so the rules can be compared after the terms take effect.

Meta’s five crawler names

Meta uses several user-agent tokens, and a robots.txt rule only applies to the token it names. Descriptions below are as summarised by the ai.robots.txt community project, which quotes Meta’s documentation where it can:

  • Meta-ExternalAgent: crawls the web “for use cases such as training AI models or improving products by indexing content directly”. Blocked on 38 sites and named on 33.
  • FacebookBot: an older token that the project says is officially used to train Meta’s speech-recognition technology. Blocked on 35.
  • Meta-ExternalFetcher: fetches individual links when a user asks a Meta AI product to. The project records that it may not follow robots.txt, because the fetch is user-initiated. Blocked on 27.
  • Meta-WebIndexer: according to Meta’s documentation, quoted by the project, it improves Meta AI search results. Named on only 10 sites and blocked on 14, mostly through catch-all rules.
  • facebookexternalhit: fetches a page’s title, image and description when a link is shared on Facebook, Instagram or WhatsApp. The project notes that blocking it stops rich link previews. Named on 12 sites, never blocked from the whole site.

What the files say

  • Meta and OpenAI are blocked by different sites. 15 sites block Meta-ExternalAgent but not GPTBot: arstechnica.com, barrons.com, ft.com, marketwatch.com, newyorker.com, nypost.com, theatlantic.com, theguardian.com, thesun.co.uk, thetimes.co.uk, theverge.com, vox.com, washingtonpost.com, wired.com, wsj.com. 11 block GPTBot but not Meta-ExternalAgent: abcnews.go.com, apnews.com, cbsnews.com, cnn.com, dallasnews.com, economist.com, latimes.com, newsnow.co.uk, newsweek.com, tampabay.com, techcrunch.com. Some publishers have licensing deals with OpenAI, which may explain part of the gap; this page does not check deals.
  • 43 of 65 sites block at least one Meta crawler other than the link-preview fetcher. 14 block all three of the newer ones, Meta-ExternalAgent, Meta-ExternalFetcher and Meta-WebIndexer.
  • The link-preview exception is deliberate on some sites. 5 sites, barrons.com, marketwatch.com, thesun.co.uk, thetimes.co.uk, wsj.com, disallow every crawler they do not name, and all 5 name facebookexternalhit so it can still read articles. Other sites leave it to the general rules.
  • Most-blocked crawlers. Common Crawl’s CCBot (50) and ClaudeBot (49) top the list, followed by Applebot-Extended (46) and PerplexityBot (44). OpenAI’s search crawler OAI-SearchBot is blocked on 20, fewer than GPTBot; OpenAI’s documentation says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers.
CrawlerOperatorWhole site disallowedof which with no Allow exceptionsNamed in file
Meta-ExternalAgentMeta382833
FacebookBotMeta352531
Meta-ExternalFetcherMeta271822
Meta-WebIndexerMeta141110
facebookexternalhitMeta0012
CCBotCommon Crawl504045
ClaudeBotAnthropic493644
Applebot-ExtendedApple463742
PerplexityBotPerplexity442839
Google-ExtendedGoogle362334
GPTBotOpenAI342242
Claude-UserAnthropic322029
ChatGPT-UserOpenAI251234
OAI-SearchBotOpenAI20830

For the wider picture across top websites, see our AI crawler census. For one publisher group’s rules in detail, see what the newspapers suing OpenAI block in robots.txt, and for the policy fight, publisher lobbying on AI crawlers.

Update, 27 October: Meta’s crawler names across a wider sample of the web

In a sample of 9,244 sites from Common Crawl’s October 2026 crawl, 4.29% of robots.txt files name Meta-ExternalAgent. That is fewer than name GPTBot (5.72%) or ClaudeBot (5.11%). Meta’s newer names are much rarer: Meta-ExternalFetcher appears in 27 files, Meta-WebIndexer in 16 and meta-externalads in 8, which is 0.09% of the sample.

Of the 397 files that name Meta-ExternalAgent, 83 disallow it from the whole site. None of the files that name meta-externalads disallow it from the whole site. By comparison, Meta-ExternalAgent is named on 33 of the 65 news sites checked above.

TokenFiles naming itShare of sampleWhole-site disallow
GPTBot5295.72%125
ClaudeBot4725.11%90
Google-Extended4534.90%67
Bytespider4384.74%106
CCBot4344.69%85
meta-externalagent3974.29%83
FacebookBot3223.48%49
facebookexternalhit1171.27%7
meta-externalfetcher270.29%9
meta-webindexer160.17%6
meta-externalads80.09%0
DeepSeekBot50.05%3

Longer-term trends in Common Crawl’s robots.txt data are in Common Crawl robots.txt denials. For a sector view, see retail robots.txt and AI agents; each token is described in the AI crawler directory.

Method

On 24 October 2026 we requested /robots.txt from 71 news domains with a plain identifying user agent. 5 did not return a file (startribune.com rate-limited us, politico.com and news.sky.com returned a challenge page, and miamiherald.com and npr.org timed out), and gannett.com was dropped because it redirects to a corporate site, leaving 65. Two files served compressed bodies and were decompressed. Following RFC 9309, for each crawler we merged every group that names its token (ignoring case) and, if none did, used the * group. A crawler counts as disallowed from the whole site when those rules include Disallow: /; the table separately shows the strict count where no Allow line carves out exceptions. philly.com and inquirer.com serve the same file and are counted separately, as are bbc.com and bbc.co.uk.

Update, 27 October. The wider sample uses 20 of the 100,000 robots.txt WARC files in Common Crawl’s CC-MAIN-2026-43 crawl. We kept responses with HTTP status 200 and the first file for each hostname, which left 9,244 hostnames. A file names a token if a User-agent line matches it exactly, ignoring case. A whole-site disallow means a group for that token contains “Disallow: /” and no Allow rules. The sample is not weighted by traffic, so it is dominated by small sites.

Caveats

  • robots.txt is a request, not access control. Nothing here says whether any crawler follows it.
  • Under Meta’s revised terms, publishers argue that a robots.txt rule may not be treated as an opt-out; that is a legal question this page does not answer.
  • This is a hand-picked list of large English-language outlets, not a sample of all news sites. Files change often; this is a single day.
  • Results for sites that were unreachable on 24 October are not included.

Download the data

Cite as: ProxyPicker, “Which News Sites Block Meta’s Crawlers”, 24 October 2026, https://proxypicker.com/news-sites-block-meta-crawlers/.

Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.