3D Model Sites and AI Crawlers: Who Blocks Them in robots.txt

Home › Guides › 3D Model Sites and AI Crawlers: Who Blocks Them in robots.txt

✓ checked 28 October 2026 · sources: our own robots.txt fetch of 29 3D model and asset sites, the ai.robots.txt list, Open3DLab, Fathom, Game Developer · method · caveats · download the data

Only 6 of 24 3D model and asset sites whose robots.txt we could read block any known AI crawler from the whole site. Three of the six are Open3DLab and its two sister sites, which say a crawler naming Roblox downloaded 145.5 GB of their files in mid-October. Sketchfab, CGTrader, Thingiverse, Printables, ArtStation, Poly Haven, the Unity Asset Store, DeviantArt and Roblox block none.

Sites checked: 29Block a known AI crawler sitewide: 6Name “trellis2”: 0

What Open3DLab says happened

Open3DLab, a free site for sharing 3D models, says that on 15 and 16 October 2026 a crawler with the user agent “trellis2-data-sourcing/1.0 (3D foundation model research; [redacted]@roblox.com)” read 18,926 project pages in 26,429 requests and then downloaded 1,027 files from 712 projects, 145.5 GB in all, across Open3DLab and two sister sites. The site says the requests came from a single Amazon Web Services address, that the crawler fetched its robots.txt and did not respect it, and that it ignored the site’s terms of service, which forbid scraping. Open3DLab says it noticed the crawler on 18 October and blocked it the same morning.

A Roblox spokesperson told Fathom on 27 October: “A Roblox researcher built an agent to collect publicly available 3D assets that mistakenly scraped content from Open3DLab, failing to operate as intended. The agent was decommissioned and the data was not used for any purpose, including training any AI models.” Game Developer reported the same statement on 28 October. Open3DLab’s account is the site’s own; we have not seen its logs.

What the robots.txt files say

A robots.txt file is a public request to crawlers about what they may fetch. It does not technically stop anyone. A site can name a crawler and disallow it, and set rules for all other crawlers under “User-agent: *”.

Open3DLab’s robots.txt, as we fetched it, names 23 user agents, including GPTBot, ClaudeBot, CCBot and img2dataset, and tells each to stay off the whole site. For every other crawler it allows the site but disallows download paths such as /project/file/download/ and /serve_file/, and asks for a two-second crawl delay. A crawler not named in the file, such as trellis2-data-sourcing, falls under those general rules. The two sister sites’ files are the same apart from the host name. We do not know what the file said on 15 October; our copy matches one we fetched at 09:00 ET on 28 October.

Blend Swap and MyMiniFactory use the same block of rules, marked in the files as Cloudflare managed content; MyMiniFactory adds a few lines of its own. Each blocks 32 named agents, 24 of them on the ai.robots.txt list, and adds a Content-Signal line with “ai-train=no”. TurboSquid blocks CCBot, GPTBot and Meta-ExternalAgent, plus TurnitinBot. None of the 24 files names trellis2, and the community ai.robots.txt list of 200 AI agents does not include it either. A named block would not have applied to an agent with a new name.

Siterobots.txtKnown AI crawlers blocked sitewideContent-Signal ai-train=no
Blend Swapread24yes
MyMiniFactoryread24yes
Open3DLabread21no
Open3DLab sister site 1read21no
Open3DLab sister site 2read21no
TurboSquidread3no
3DExportread0no
ArtStationread0no
BOOTHread0no
CGTraderread0no
Daz 3Dread0no
DeviantArtread0no
MakerWorldread0no
Poly Havenread0no
Printablesread0no
RenderHubread0no
Robloxread0no
Roblox Creator Hubread0no
Sketchfabread0no
Thangsread0no
The Models Resourceread0no
Thingiverseread0no
Unity Asset Storeread0no
VRoid Hubread0no
Cultsrefused (403)
Fabrefused (403)
Free3Drefused (403)
Unreal Enginerefused (403)
VRoidno file (404)

4 sites refused our robots.txt request with a 403 error, and one had no robots.txt. We list the two sister sites without names. For which AI crawlers exist and what they do, see the AI crawler directory. For copyright cases over AI training data, see the AI copyright lawsuit tracker, and for the law on scraping, the scraping law hub.

Method

On 28 October 2026 at 1:44 PM ET we sent one request to /robots.txt on each host with a user agent naming ProxyPicker, following redirects as the robots.txt standard (RFC 9309) allows. For each file we listed the named user agents that are disallowed from the root path, using longest-match rules, and checked them against the ai.robots.txt project’s list of AI agents, fetched the same day. Where a site has both a www and a bare host, we counted it once.

Caveats

  • robots.txt is voluntary. A site may also block crawlers with a firewall or terms of service, which this check does not see.
  • A 403 error to our request may mean the site blocks unknown automated clients, not that it has no robots.txt.
  • The incident figures come from Open3DLab and the reports that cite it; Roblox has confirmed the scraping but not the figures.

Download the data

Cite as: ProxyPicker, “3D Model Sites and AI Crawlers: Who Blocks Them in robots.txt”, 28 October 2026, https://proxypicker.com/3d-model-sites-ai-crawlers/.

Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.