Which AI Companies Block Rivals’ Crawlers

Home › Guides › AI Companies and Rival Crawlers

✓ checked 20 October 2026 · source: our fetch of robots.txt from 50 AI-company websites on 20 October 2026 · the sites · method · download the data

AI companies send crawlers across the web, and some of them tell rival companies’ crawlers to stay off their own sites. We read the robots.txt files of 50 websites run by AI labs and AI-agent companies. 47 could be read. Of those, 6 block at least one AI crawler or agent run by another company: ChatGPT’s, Claude’s, Grok’s and Meta AI’s chat sites, Together AI and Amazon. The companies’ main corporate sites, such as openai.com and anthropic.com, block none.

6 of 47 block a rival’s AI bot6 of those block Google-Extended39 AI bots blocked by Amazon38 name no AI bot at all

What robots.txt does here

A robots.txt file is a plain-text list of rules at the root of a website. Each group of rules starts with a user agent, the name a crawler announces, and says which paths it may not fetch. AI companies publish names for their crawlers, such as OpenAI’s GPTBot and Anthropic’s ClaudeBot, and say they follow these rules. A few names, such as Google-Extended and Applebot-Extended, aren’t crawlers at all: they are tokens that tell Google or Apple not to use a site’s content for AI. Robots.txt is a request, not a lock. It only works if the crawler names itself and chooses to comply, and it doesn’t stop a person or an agent using an ordinary browser. For the names and what each one does, see our AI crawler directory.

The question matters now because the same companies are arguing about who may crawl whom. In October, Nieman Lab reported that Meta’s new commercial terms, due to take effect on 30 October, would let it crawl websites “associated with” business accounts on its platforms, and a Meta spokesperson said that “content owners can still manage how their content is accessed by using robots.txt.” Here is how AI companies use robots.txt on their own sites.

Who blocks whom

SiteOwnerRival AI bots blockedHow muchWhich ones, by operator
chatgpt.comOpenAI12Whole siteAnthropic: anthropic-ai, Claude-Web; ByteDance: Bytespider; Common Crawl: CCBot; Meta: FacebookBot; Google: Google-Extended; open-source tool: img2dataset; Brandwatch: magpie-crawler; Webz.io: Omgili, Omgilibot; Perplexity: PerplexityBot, Perplexity‑User
claude.aiAnthropic4Whole siteOpenAI: ChatGPT-User, GPTBot, OAI-SearchBot; Google: Google-Extended
grok.comxAI6All but the homepage and a few pagesApple: Applebot-Extended; OpenAI: ChatGPT-User, GPTBot; Anthropic: ClaudeBot; Google: Google-Extended; Perplexity: PerplexityBot
meta.aiMeta6Whole siteAmazon: Amazonbot; Apple: Applebot-Extended; Anthropic: ClaudeBot; Google: Google-Extended; OpenAI: GPTBot; Perplexity: PerplexityBot
together.aiTogether AI1Whole siteGoogle: Google-Extended
amazon.comAmazon39Whole siteRivals’ crawlers and agents from 22 operators, including OpenAI, Anthropic, Google, Meta, Perplexity, Mistral AI and DeepSeek

“Blocked” means the robots.txt group for that name disallows ordinary pages. Names are as written in each file. Full lists are in the CSV.

  • chatgpt.com names 12 AI bots and blocks all of them from the whole site, including Anthropic’s older anthropic-ai and Claude-Web names, Perplexity’s crawler, ByteDance’s Bytespider, Common Crawl’s CCBot and Google-Extended. Its catch-all group also blocks ordinary pages for any crawler it doesn’t list. One entry, Perplexity’s user agent, is written with a non-standard hyphen character, so a crawler matching its name exactly wouldn’t find it.
  • claude.ai blocks OpenAI’s three agents (GPTBot, OAI-SearchBot and ChatGPT-User) and Google-Extended from the whole site.
  • grok.com, xAI’s chat site, gives GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and Applebot-Extended one rule group: the homepage and a handful of marketing pages are allowed, everything else is disallowed.
  • meta.ai blocks GPTBot, ClaudeBot, PerplexityBot, Amazonbot, Google-Extended and Applebot-Extended, along with several non-AI crawlers. Its file opens with a notice that automated collection is prohibited without Meta’s written permission, and its catch-all group blocks the whole site for unlisted crawlers.
  • amazon.com has the longest list: it blocks 39 AI crawlers and agents that we recognise, from 22 operators. We compared Amazon’s file with those of other retailers in Which US Retailers Block AI Shopping Agents.
  • together.ai blocks only Google-Extended.

Sites that name AI bots without blocking them

3 more sites name AI bots but don’t keep them out. stability.ai lists 23 AI crawlers in the same group as its catch-all rules, so they get the same access as everyone else. poe.com does the reverse of a block: its catch-all group disallows the whole site, while GPTBot and Google-Extended are kept only from pages under /p/, so they can reach more than an unnamed crawler. lovable.dev names nine AI bots to give them rules that differ slightly from other crawlers’.

The other 38 readable files name no AI bot at all, among them openai.com, anthropic.com, perplexity.ai, mistral.ai, x.ai, gemini.google.com and huggingface.co. Four sites, x.ai, elevenlabs.io, firecrawl.dev and nvidia.com, go further and add a Content-Signal line saying their content may be used for AI training, search and AI answers (ai-train=yes). For how widely that line is used, see our robots.txt AI-preferences survey.

What this doesn’t show

  • We checked one robots.txt per domain, on one day. Companies also run other domains, apps and APIs.
  • The list of sites is our own choice of well-known AI labs, AI apps and agent tools, not a complete or random sample.
  • robots.txt states a preference. Whether a crawler follows it, and whether a site also blocks crawlers at its servers, isn’t visible in the file.
  • Many entries in these files are old names that companies have since replaced, so a block can be partly symbolic.

Method

On 20 October 2026 we requested /robots.txt from 50 domains with a user agent identifying ProxyPicker, retrying once. A file counted as readable if it returned HTTP 200, wasn’t an HTML page and contained a user-agent line. midjourney.com returned 403, and moonshot.ai and qwen.ai returned web pages instead of robots.txt. amazon.com failed twice with a connection error and was fetched on a later retry. We parsed each file under RFC 9309, the robots.txt standard: a crawler uses the group that names it, or the catch-all group if none does, and the longest matching rule wins. A name counts as blocked if its rules disallow a typical page path, and as blocked from the whole site if they also disallow /. We counted a name as an AI crawler or agent if it appears in our AI crawler directory or is a known AI name such as anthropic-ai, Bytespider, cohere-ai or Diffbot; non-AI crawlers such as PetalBot, Yandex or Baiduspider aren’t counted. A block is a rival’s if the bot’s operator isn’t the site’s owner. A separate script re-parses every file and recounts each number.

Download the data

  • AI-company robots.txt files (CSV): all 50 domains, with owner, readability, the number of names, AI names, AI names blocked, rival names blocked and any Content-Signal line.
  • Every AI bot named (CSV): one row per site and AI user agent, with its operator and whether and how much it is blocked.

Cite as: ProxyPicker, “Which AI Companies Block Rivals’ Crawlers”, 20 October 2026, https://proxypicker.com/ai-companies-block-rival-crawlers/.

Related: AI-Crawler & llms.txt Census · Retailers and AI shopping agents · AI crawler directory.

Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.