Which US Retailers Block AI Shopping Agents

Home › Guides › Retail robots.txt vs AI Agents

✓ checked 19 October 2026 · source: robots.txt files fetched 19 October 2026 from 50 large US retail, marketplace and delivery sites · sites · agents · method · download the data

Of 33 big US retail and delivery sites whose robots.txt we could read, 2 tell AI agents that act for a user, such as ChatGPT-User or Perplexity-User, to stay off the whole site: Amazon and Instacart. Amazon names 99 different crawlers and agents in its file, more than any other site checked. 21 of the 33 sites, including Walmart, Target, Costco and Home Depot, don’t mention any AI crawler or agent at all.

33 robots.txt files read2 block user-triggered AI agents4 block any AI training crawler21 name no AI agent

Why this is in the news

In September Amazon cut off Muse, Meta’s new personal AI agent, from shopping on Amazon.com, GeekWire reported on 20 September. Amazon told GeekWire that Meta hadn’t told it Muse would use its store, that the agent “doesn’t identify itself when it browses”, and that it appeared to capture and store customer credentials. Meta had said Muse “has no visibility into people’s passwords or payment methods”. Muse users saw a message saying that “Continued access by an unauthorized AI agent violates Amazon’s Conditions of Use”. GeekWire also reported that Amazon has been moving to block shopping agents from Google and OpenAI.

Amazon didn’t use robots.txt against Muse, and it couldn’t have. Robots.txt only works when a crawler announces a name (its user agent) and chooses to follow the rules for that name. An agent that browses through the customer’s own logged-in browser session has no separate name to block. Meta’s commercial terms raised a related question in October: Nieman Lab reported that new terms, due to take effect on 30 October, would let Meta crawl websites “associated with” business accounts on its platforms. A Meta spokesperson told Nieman Lab that “content owners can still manage how their content is accessed by using robots.txt.” Meta later told TechCrunch that the crawling terms “only applied to our ads and business products” and that the change “does not change how Meta uses web crawlers”.

Three kinds of AI visitor

AI companies run different bots for different jobs, and a site can treat each differently. Our AI crawler directory sorts the documented ones into groups:

  • Training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent, Amazonbot and others) collect pages that may be used to train models.
  • AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot and others) build indexes for AI search and answer features.
  • User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher and five others) visit a page when a person asks the assistant to, for example to compare prices. Several operators say these fetchers may not follow robots.txt because a person started the request.

For a retailer the trade-off differs by group. Blocking training crawlers keeps product pages out of training data. Blocking user-triggered fetchers can stop an assistant from reading a product page when a shopper asks about it, which can mean lost sales.

What the 12 sites that mention AI agents say

SiteUser agents namedAI agents namedAI agents blocked from the whole siteCatch-all rule blocks the whole site
amazon.com99265 user-triggered, 6 training, 5 search, 7 otherno
bestbuy.com3212noneno
ebay.com20125 training, 1 searchno
ulta.com1610noneno
temu.com329noneno
gap.com128noneno
instacart.com2168 user-triggered, 9 training, 7 search, 7 otheryes
cvs.com125noneno
overstock.com84noneno
nordstrom.com52noneno
petsmart.com42noneno
chewy.com311 trainingno

“Blocked from the whole site” means the rules that apply to that agent disallow the home page (/), under RFC 9309 matching. That includes agents covered only by a catch-all (*) rule. “Other” means agent names that aren’t in our directory, such as GoogleAgent-Shopping. Sites that name an agent but only block some paths, such as checkout or search pages, aren’t counted as blocking it.

  • Amazon has 100 user-agent lines covering 99 distinct names (ClaudeBot appears twice). It blocks the whole site to 5 of the 9 user-triggered fetchers in our directory (ChatGPT-User, Claude-User, meta-externalfetcher, Perplexity-User and MistralAI-User), to 6 training crawlers and 5 AI search crawlers. It also blocks 7 agent names we couldn’t match to operator documentation, including GoogleAgent-Shopping and GoogleAgent-Mariner. It doesn’t block its own Amazonbot.
  • Instacart works the other way round: it lists the bots it allows and ends with a rule that blocks all others. ChatGPT-User, OAI-SearchBot, Googlebot and Applebot are allowed, with some paths excluded. Perplexity-User and PerplexityBot are blocked by name, and the catch-all rule covers Claude-User, meta-externalfetcher and the other unnamed agents. In total Instacart blocks 8 of the 9 user-triggered fetchers from the whole site; ChatGPT-User is the only one it lets in.
  • eBay and Chewy block training crawlers but not user-triggered fetchers. eBay blocks GPTBot, ClaudeBot, CCBot, meta-externalagent and Amazonbot, and gives ChatGPT-User and Claude-User a long list of disallowed paths. Chewy blocks only one AI crawler, Amazonbot.
  • ChatGPT-User is named by 8 sites, but only Amazon blocks it from the whole site. The other 7 (Best Buy, CVS, eBay, Gap, Instacart, Temu and Ulta) name it only to set rules for particular paths, such as account, API or search pages.

The 21 sites that name no AI crawler or agent are: walmart.com, target.com, homedepot.com, costco.com, etsy.com, wayfair.com, shein.com, apple.com, nike.com, ikea.com, walgreens.com, doordash.com, samsclub.com, zappos.com, newegg.com, williams-sonoma.com, shopify.com, bhphotovideo.com, bathandbodyworks.com, victoriassecret.com, dollargeneral.com. Their rules for all bots apply to AI agents too, and none of them blocks the whole site with a catch-all rule.

The user-triggered fetchers

AgentOperatorWhat the operator says about robots.txtSites naming itSites blocking it from the whole site
ChatGPT-UserOpenAIMay not follow (user-initiated)8amazon.com
Claude-UserAnthropicYes: says it follows robots.txt5amazon.com, instacart.com
Google-AgentGoogleMay not follow (user-initiated)0instacart.com
Google-GeminiNotebookGoogleMay not follow (user-initiated)0instacart.com
meta-externalfetcherMetaMay not follow (user-initiated)1amazon.com, instacart.com
Perplexity-UserPerplexityMay not follow (user-initiated)3amazon.com, instacart.com
Amzn-UserAmazonMay not follow (user-initiated)0instacart.com
MistralAI-UserMistral AIOpt-out via robots.txt documented1amazon.com, instacart.com
Kimi-UserMoonshot AI (Kimi)May not follow (user-initiated)0instacart.com

Operator statements are from our AI crawler directory, which links each operator’s documentation.

What robots.txt can’t show

  • Robots.txt is a request, not a lock. Many retailers also block bots on their servers, through anti-bot services that look at IP addresses and browser behaviour. We couldn’t read the robots.txt files of 17 of the 50 sites because they refused or didn’t answer our clearly labelled research request (10 returned HTTP 403): lowes.com, macys.com, kohls.com, aliexpress.us, sephora.com, kroger.com, qvc.com, dickssportinggoods.com, staples.com, officedepot.com, rei.com, adidas.com, hm.com, zara.com, uniqlo.com, lululemon.com, jcpenney.com. For how common this kind of filtering is, see our Anti-Bot Census.
  • Agents that use a customer’s browser can’t be named. As the Muse case shows, these agents can only be blocked by detecting them, through terms of service, or by asking the AI company to exclude the site.
  • The list is hand-picked. It covers 50 large US retail, marketplace and delivery sites, not a random sample. Results for other retailers may differ.

Method

On 19 October 2026 we requested /robots.txt from www. and the bare domain of each of the 50 sites, with a user agent identifying ProxyPicker research. A file counts as readable if it returned HTTP 200 with at least one User-agent line. We parsed each file under RFC 9309: an agent uses the group that names its product token (ignoring any version suffix such as /1.1), or the * group if none names it, and the longest matching rule wins, with Allow winning ties. We checked each of the 36 agents and control tokens in our AI crawler directory, plus 7 other agent names found in the files, against the home-page path. We didn’t use proxies or retry with a different user agent.

Download the data

  • By site (CSV): all 50 sites, fetch result, user agents named, and AI agents blocked from the whole site by category.
  • By agent (CSV): each AI agent, its operator and category, how many sites name it and which block it.

Cite as: ProxyPicker, “Which US Retailers Block AI Shopping Agents in robots.txt”, 19 October 2026, https://proxypicker.com/retail-robots-txt-ai-agents/.

Related: AI crawler directory · AI-Crawler & llms.txt Census · Which federal .gov sites block AI bots · Anti-Bot Census.

Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.