Home › Guides › Airlines, Banks and Travel Sites: What Their robots.txt Says to AI Agents
✓ checked 27 October 2026 · source: each site’s own robots.txt, fetched 27 October 2026 · method · caveats · download the data
None of the 13 airline robots.txt files we could read names a single AI crawler or AI agent, and only 2 of 15 bank and broker files do. Delta, JPMorgan Chase and Charles Schwab have each been reported in October as keeping AI agents out, but their robots.txt files contain only a general rule for all bots. Travel sites are different: 8 of 14 name AI bots, usually to keep them out of selected pages.
Readable files: 55Name ChatGPT-User: 12Block ChatGPT-User from the whole site: 0
Why this matters
The Verge reported on 17 October that sites including Delta, JPMorgan Chase and Charles Schwab are blocking AI agents, and that Yelp told TechCrunch it only allows agents that pay for its data licence. robots.txt is the public file where a site can ask named bots to stay out. These results show that, for the airlines and banks we checked, any blocking of agents is not being announced there; it would happen in login systems, terms of service or bot-detection tools.
By sector
| Sector | Sites checked | Readable robots.txt | Name any AI bot |
|---|---|---|---|
| Airlines | 15 | 13 | 0 |
| Travel and hotels | 15 | 14 | 8 |
| Banks and brokers | 15 | 15 | 2 |
| Marketplaces | 15 | 13 | 6 |
The agents that act for users
ChatGPT-User, Claude-User and Perplexity-User are the names OpenAI, Anthropic and Perplexity give to fetches made when a person asks their assistant to visit a page. ChatGPT-User appears in 12 of the 55 files, Claude-User in 6 and Perplexity-User in 6. No file blocks ChatGPT-User or Claude-User from the whole site; the rules that name them shut off selected paths, such as search results or checkout, on sites like Expedia, Airbnb, eBay and Instacart. The training crawler meta-externalagent is blocked from the whole site by 4 sites.
Similar results for shops are in retail robots.txt and AI agents. Agent names and their operators are listed in the AI crawler directory, and the tools used to get past bot checks in the anti-bot bypass economy.
Site-by-site results
| Sector | Site | ChatGPT-User | Claude-User | Perplexity-User | GPTBot | meta-externalagent |
|---|---|---|---|---|---|---|
| Airlines | delta.com | not named | not named | not named | not named | not named |
| Airlines | united.com | not named | not named | not named | not named | not named |
| Airlines | aa.com | not named | not named | not named | not named | not named |
| Airlines | southwest.com | not named | not named | not named | not named | not named |
| Airlines | jetblue.com | not named | not named | not named | not named | not named |
| Airlines | alaskaair.com | not named | not named | not named | not named | not named |
| Airlines | britishairways.com | not named | not named | not named | not named | not named |
| Airlines | lufthansa.com | not named | not named | not named | not named | not named |
| Airlines | emirates.com | not named | not named | not named | not named | not named |
| Airlines | qatarairways.com | not named | not named | not named | not named | not named |
| Airlines | ryanair.com | not named | not named | not named | not named | not named |
| Airlines | easyjet.com | not named | not named | not named | not named | not named |
| Airlines | singaporeair.com | not named | not named | not named | not named | not named |
| Travel and hotels | amtrak.com | not named | not named | not named | not named | not named |
| Travel and hotels | expedia.com | partial | partial | partial | not named | not named |
| Travel and hotels | booking.com | not named | not named | not named | not named | not named |
| Travel and hotels | kayak.com | partial | not named | not named | not named | not named |
| Travel and hotels | tripadvisor.com | partial | not named | not named | blocked | blocked |
| Travel and hotels | airbnb.com | partial | not named | not named | blocked | partial |
| Travel and hotels | vrbo.com | partial | partial | partial | not named | not named |
| Travel and hotels | hotels.com | partial | partial | partial | not named | not named |
| Travel and hotels | priceline.com | not named | not named | not named | not named | not named |
| Travel and hotels | marriott.com | not named | not named | not named | not named | not named |
| Travel and hotels | hilton.com | not named | not named | not named | not named | not named |
| Travel and hotels | ihg.com | not named | not named | not named | not named | not named |
| Travel and hotels | skyscanner.net | not named | not named | not named | not named | blocked |
| Travel and hotels | google.com | not named | not named | not named | not named | not named |
| Banks and brokers | chase.com | not named | not named | not named | not named | not named |
| Banks and brokers | bankofamerica.com | not named | not named | not named | not named | not named |
| Banks and brokers | wellsfargo.com | not named | not named | not named | not named | not named |
| Banks and brokers | citi.com | not named | not named | not named | not named | not named |
| Banks and brokers | capitalone.com | not named | not named | not named | not named | not named |
| Banks and brokers | usbank.com | not named | not named | not named | not named | not named |
| Banks and brokers | schwab.com | not named | not named | not named | not named | not named |
| Banks and brokers | fidelity.com | not named | not named | not named | not named | not named |
| Banks and brokers | vanguard.com | not named | not named | not named | not named | not named |
| Banks and brokers | americanexpress.com | not named | not named | not named | not named | not named |
| Banks and brokers | discover.com | not named | not named | not named | not named | not named |
| Banks and brokers | paypal.com | not named | not named | not named | not named | not named |
| Banks and brokers | robinhood.com | not named | not named | not named | not named | not named |
| Banks and brokers | morganstanley.com | allowed | not named | not named | allowed | allowed |
| Banks and brokers | goldmansachs.com | partial | not named | not named | partial | not named |
| Marketplaces | ebay.com | partial | partial | not named | partial | partial |
| Marketplaces | etsy.com | not named | not named | not named | not named | not named |
| Marketplaces | craigslist.org | not named | not named | not named | not named | not named |
| Marketplaces | zillow.com | not named | not named | not named | not named | partial |
| Marketplaces | realtor.com | not named | not named | not named | not named | blocked |
| Marketplaces | opentable.com | partial | partial | partial | not named | not named |
| Marketplaces | ticketmaster.com | not named | not named | not named | not named | not named |
| Marketplaces | stubhub.com | not named | not named | not named | not named | not named |
| Marketplaces | doordash.com | not named | not named | not named | not named | not named |
| Marketplaces | ubereats.com | not named | not named | not named | not named | not named |
| Marketplaces | instacart.com | partial | not named | blocked | not named | blocked |
| Marketplaces | autotrader.com | partial | partial | partial | partial | not named |
| Marketplaces | carvana.com | not named | not named | not named | not named | not named |
“blocked” means Disallow: / for that name; “partial” means some paths are disallowed; “not named” means only the general rules apply.
Method
We fetched /robots.txt from 60 airline, travel, bank and marketplace sites on 27 October 2026, retrying the www host and a browser user agent when the first request failed. 5 sites returned an error, a block page or a page without robots rules (airfrance.com, aircanada.com, hyatt.com, yelp.com, cars.com) and are left out. For each file we grouped the rules by user agent and checked 15 AI crawler and agent names. Full results for all 15 names are in the CSV.
Caveats
- robots.txt is a request, not an enforcement tool. A site can block agents through other means while its robots.txt says nothing about them.
- Yelp, which the Verge report says limits agents to paying licensees, returned an error to our request and is not included.
- The site list is our own selection of large US and European brands.
Download the data
Cite as: ProxyPicker, “Airlines, Banks and Travel Sites: What Their robots.txt Says to AI Agents”, 27 October 2026, https://proxypicker.com/travel-bank-robots-txt-ai-agents/.
Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.