Home › Guides › 3D Model Sites and AI Crawlers: Who Blocks Them in robots.txt
✓ checked 28 October 2026 · sources: our own robots.txt fetch of 29 3D model and asset sites, the ai.robots.txt list, Open3DLab, Fathom, Game Developer · method · caveats · download the data
Only 6 of 24 3D model and asset sites whose robots.txt we could read block any known AI crawler from the whole site. Three of the six are Open3DLab and its two sister sites, which say a crawler naming Roblox downloaded 145.5 GB of their files in mid-October. Sketchfab, CGTrader, Thingiverse, Printables, ArtStation, Poly Haven, the Unity Asset Store, DeviantArt and Roblox block none.
Sites checked: 29Block a known AI crawler sitewide: 6Name “trellis2”: 0
What Open3DLab says happened
Open3DLab, a free site for sharing 3D models, says that on 15 and 16 October 2026 a crawler with the user agent “trellis2-data-sourcing/1.0 (3D foundation model research; [redacted]@roblox.com)” read 18,926 project pages in 26,429 requests and then downloaded 1,027 files from 712 projects, 145.5 GB in all, across Open3DLab and two sister sites. The site says the requests came from a single Amazon Web Services address, that the crawler fetched its robots.txt and did not respect it, and that it ignored the site’s terms of service, which forbid scraping. Open3DLab says it noticed the crawler on 18 October and blocked it the same morning.
A Roblox spokesperson told Fathom on 27 October: “A Roblox researcher built an agent to collect publicly available 3D assets that mistakenly scraped content from Open3DLab, failing to operate as intended. The agent was decommissioned and the data was not used for any purpose, including training any AI models.” Game Developer reported the same statement on 28 October. Open3DLab’s account is the site’s own; we have not seen its logs.
What the robots.txt files say
A robots.txt file is a public request to crawlers about what they may fetch. It does not technically stop anyone. A site can name a crawler and disallow it, and set rules for all other crawlers under “User-agent: *”.
Open3DLab’s robots.txt, as we fetched it, names 23 user agents, including GPTBot, ClaudeBot, CCBot and img2dataset, and tells each to stay off the whole site. For every other crawler it allows the site but disallows download paths such as /project/file/download/ and /serve_file/, and asks for a two-second crawl delay. A crawler not named in the file, such as trellis2-data-sourcing, falls under those general rules. The two sister sites’ files are the same apart from the host name. We do not know what the file said on 15 October; our copy matches one we fetched at 09:00 ET on 28 October.
Blend Swap and MyMiniFactory use the same block of rules, marked in the files as Cloudflare managed content; MyMiniFactory adds a few lines of its own. Each blocks 32 named agents, 24 of them on the ai.robots.txt list, and adds a Content-Signal line with “ai-train=no”. TurboSquid blocks CCBot, GPTBot and Meta-ExternalAgent, plus TurnitinBot. None of the 24 files names trellis2, and the community ai.robots.txt list of 200 AI agents does not include it either. A named block would not have applied to an agent with a new name.
| Site | robots.txt | Known AI crawlers blocked sitewide | Content-Signal ai-train=no |
|---|---|---|---|
| Blend Swap | read | 24 | yes |
| MyMiniFactory | read | 24 | yes |
| Open3DLab | read | 21 | no |
| Open3DLab sister site 1 | read | 21 | no |
| Open3DLab sister site 2 | read | 21 | no |
| TurboSquid | read | 3 | no |
| 3DExport | read | 0 | no |
| ArtStation | read | 0 | no |
| BOOTH | read | 0 | no |
| CGTrader | read | 0 | no |
| Daz 3D | read | 0 | no |
| DeviantArt | read | 0 | no |
| MakerWorld | read | 0 | no |
| Poly Haven | read | 0 | no |
| Printables | read | 0 | no |
| RenderHub | read | 0 | no |
| Roblox | read | 0 | no |
| Roblox Creator Hub | read | 0 | no |
| Sketchfab | read | 0 | no |
| Thangs | read | 0 | no |
| The Models Resource | read | 0 | no |
| Thingiverse | read | 0 | no |
| Unity Asset Store | read | 0 | no |
| VRoid Hub | read | 0 | no |
| Cults | refused (403) | ||
| Fab | refused (403) | ||
| Free3D | refused (403) | ||
| Unreal Engine | refused (403) | ||
| VRoid | no file (404) |
4 sites refused our robots.txt request with a 403 error, and one had no robots.txt. We list the two sister sites without names. For which AI crawlers exist and what they do, see the AI crawler directory. For copyright cases over AI training data, see the AI copyright lawsuit tracker, and for the law on scraping, the scraping law hub.
Method
On 28 October 2026 at 1:44 PM ET we sent one request to /robots.txt on each host with a user agent naming ProxyPicker, following redirects as the robots.txt standard (RFC 9309) allows. For each file we listed the named user agents that are disallowed from the root path, using longest-match rules, and checked them against the ai.robots.txt project’s list of AI agents, fetched the same day. Where a site has both a www and a bare host, we counted it once.
Caveats
- robots.txt is voluntary. A site may also block crawlers with a firewall or terms of service, which this check does not see.
- A 403 error to our request may mean the site blocks unknown automated clients, not that it has no robots.txt.
- The incident figures come from Open3DLab and the reports that cite it; Roblox has confirmed the scraping but not the figures.
Download the data
Cite as: ProxyPicker, “3D Model Sites and AI Crawlers: Who Blocks Them in robots.txt”, 28 October 2026, https://proxypicker.com/3d-model-sites-ai-crawlers/.
Press & data requests: email [email protected] for interviews, the raw data behind this page, or corrections. See also our contact page.