Airlines, Banks and Travel Sites: What Their robots.txt Says to AI Agents

Home › Guides › Airlines, Banks and Travel Sites: What Their robots.txt Says to AI Agents

✓ checked 27 October 2026 · source: each site’s own robots.txt, fetched 27 October 2026 · method · caveats · download the data

None of the 13 airline robots.txt files we could read names a single AI crawler or AI agent, and only 2 of 15 bank and broker files do. Delta, JPMorgan Chase and Charles Schwab have each been reported in October as keeping AI agents out, but their robots.txt files contain only a general rule for all bots. Travel sites are different: 8 of 14 name AI bots, usually to keep them out of selected pages.

Readable files: 55Name ChatGPT-User: 12Block ChatGPT-User from the whole site: 0

Why this matters

The Verge reported on 17 October that sites including Delta, JPMorgan Chase and Charles Schwab are blocking AI agents, and that Yelp told TechCrunch it only allows agents that pay for its data licence. robots.txt is the public file where a site can ask named bots to stay out. These results show that, for the airlines and banks we checked, any blocking of agents is not being announced there; it would happen in login systems, terms of service or bot-detection tools.

Readable robots.txt files that name at least one AI crawler or agentAirlinesAirlines: 00Travel and hotelsTravel and hotels: 88Banks and brokersBanks and brokers: 22MarketplacesMarketplaces: 66
Number of readable robots.txt files in each sector that contain a rule for at least one of 15 AI crawler or agent names. Fetched 27 October 2026.

By sector

SectorSites checkedReadable robots.txtName any AI bot
Airlines15130
Travel and hotels15148
Banks and brokers15152
Marketplaces15136

The agents that act for users

ChatGPT-User, Claude-User and Perplexity-User are the names OpenAI, Anthropic and Perplexity give to fetches made when a person asks their assistant to visit a page. ChatGPT-User appears in 12 of the 55 files, Claude-User in 6 and Perplexity-User in 6. No file blocks ChatGPT-User or Claude-User from the whole site; the rules that name them shut off selected paths, such as search results or checkout, on sites like Expedia, Airbnb, eBay and Instacart. The training crawler meta-externalagent is blocked from the whole site by 4 sites.

Similar results for shops are in retail robots.txt and AI agents. Agent names and their operators are listed in the AI crawler directory, and the tools used to get past bot checks in the anti-bot bypass economy.

Site-by-site results

SectorSiteChatGPT-UserClaude-UserPerplexity-UserGPTBotmeta-externalagent
Airlinesdelta.comnot namednot namednot namednot namednot named
Airlinesunited.comnot namednot namednot namednot namednot named
Airlinesaa.comnot namednot namednot namednot namednot named
Airlinessouthwest.comnot namednot namednot namednot namednot named
Airlinesjetblue.comnot namednot namednot namednot namednot named
Airlinesalaskaair.comnot namednot namednot namednot namednot named
Airlinesbritishairways.comnot namednot namednot namednot namednot named
Airlineslufthansa.comnot namednot namednot namednot namednot named
Airlinesemirates.comnot namednot namednot namednot namednot named
Airlinesqatarairways.comnot namednot namednot namednot namednot named
Airlinesryanair.comnot namednot namednot namednot namednot named
Airlineseasyjet.comnot namednot namednot namednot namednot named
Airlinessingaporeair.comnot namednot namednot namednot namednot named
Travel and hotelsamtrak.comnot namednot namednot namednot namednot named
Travel and hotelsexpedia.compartialpartialpartialnot namednot named
Travel and hotelsbooking.comnot namednot namednot namednot namednot named
Travel and hotelskayak.compartialnot namednot namednot namednot named
Travel and hotelstripadvisor.compartialnot namednot namedblockedblocked
Travel and hotelsairbnb.compartialnot namednot namedblockedpartial
Travel and hotelsvrbo.compartialpartialpartialnot namednot named
Travel and hotelshotels.compartialpartialpartialnot namednot named
Travel and hotelspriceline.comnot namednot namednot namednot namednot named
Travel and hotelsmarriott.comnot namednot namednot namednot namednot named
Travel and hotelshilton.comnot namednot namednot namednot namednot named
Travel and hotelsihg.comnot namednot namednot namednot namednot named
Travel and hotelsskyscanner.netnot namednot namednot namednot namedblocked
Travel and hotelsgoogle.comnot namednot namednot namednot namednot named
Banks and brokerschase.comnot namednot namednot namednot namednot named
Banks and brokersbankofamerica.comnot namednot namednot namednot namednot named
Banks and brokerswellsfargo.comnot namednot namednot namednot namednot named
Banks and brokersciti.comnot namednot namednot namednot namednot named
Banks and brokerscapitalone.comnot namednot namednot namednot namednot named
Banks and brokersusbank.comnot namednot namednot namednot namednot named
Banks and brokersschwab.comnot namednot namednot namednot namednot named
Banks and brokersfidelity.comnot namednot namednot namednot namednot named
Banks and brokersvanguard.comnot namednot namednot namednot namednot named
Banks and brokersamericanexpress.comnot namednot namednot namednot namednot named
Banks and brokersdiscover.comnot namednot namednot namednot namednot named
Banks and brokerspaypal.comnot namednot namednot namednot namednot named
Banks and brokersrobinhood.comnot namednot namednot namednot namednot named
Banks and brokersmorganstanley.comallowednot namednot namedallowedallowed
Banks and brokersgoldmansachs.compartialnot namednot namedpartialnot named
Marketplacesebay.compartialpartialnot namedpartialpartial
Marketplacesetsy.comnot namednot namednot namednot namednot named
Marketplacescraigslist.orgnot namednot namednot namednot namednot named
Marketplaceszillow.comnot namednot namednot namednot namedpartial
Marketplacesrealtor.comnot namednot namednot namednot namedblocked
Marketplacesopentable.compartialpartialpartialnot namednot named
Marketplacesticketmaster.comnot namednot namednot namednot namednot named
Marketplacesstubhub.comnot namednot namednot namednot namednot named
Marketplacesdoordash.comnot namednot namednot namednot namednot named
Marketplacesubereats.comnot namednot namednot namednot namednot named
Marketplacesinstacart.compartialnot namedblockednot namedblocked
Marketplacesautotrader.compartialpartialpartialpartialnot named
Marketplacescarvana.comnot namednot namednot namednot namednot named

“blocked” means Disallow: / for that name; “partial” means some paths are disallowed; “not named” means only the general rules apply.

Method

We fetched /robots.txt from 60 airline, travel, bank and marketplace sites on 27 October 2026, retrying the www host and a browser user agent when the first request failed. 5 sites returned an error, a block page or a page without robots rules (airfrance.com, aircanada.com, hyatt.com, yelp.com, cars.com) and are left out. For each file we grouped the rules by user agent and checked 15 AI crawler and agent names. Full results for all 15 names are in the CSV.

Caveats

  • robots.txt is a request, not an enforcement tool. A site can block agents through other means while its robots.txt says nothing about them.
  • Yelp, which the Verge report says limits agents to paying licensees, returned an error to our request and is not included.
  • The site list is our own selection of large US and European brands.

Download the data

Cite as: ProxyPicker, “Airlines, Banks and Travel Sites: What Their robots.txt Says to AI Agents”, 27 October 2026, https://proxypicker.com/travel-bank-robots-txt-ai-agents/.

Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.