NeuralCrawl

Roblox / robots.txt

← back to roblox.com · change detected 14d ago (2026-07-21T19:32:39Z) · modified +61 −0

What changed

Diff old (2026-07-21T01:14:44Z) → new (2026-07-21T19:32:39Z)

@@ -1,3 +1,64 @@
1+ # As a condition of accessing this website, you agree to abide by the following
2+ # content signals:
3+
4+ # (a) If a Content-Signal = yes, you may collect content for the corresponding
5+ # use.
6+ # (b) If a Content-Signal = no, you may not collect content for the
7+ # corresponding use.
8+ # (c) If the website operator does not include a Content-Signal for a
9+ # corresponding use, the website operator neither grants nor restricts
10+ # permission via Content-Signal with respect to the corresponding use.
11+
12+ # The content signals and their meanings are:
13+
14+ # search: building a search index and providing search results (e.g., returning
15+ # hyperlinks and short excerpts from your website's contents). Search does not
16+ # include providing AI-generated search summaries.
17+ # ai-input: inputting content into one or more AI models (e.g., retrieval
18+ # augmented generation, grounding, or other real-time taking of content for
19+ # generative AI search answers).
20+ # ai-train: training or fine-tuning AI models.
21+ # use: how AI systems may consume the content (immediate, reference, or full).
22+
23+ # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
24+ # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
25+ # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
26+
27+ # BEGIN Cloudflare Managed content
28+
29+ User-agent: *
30+ Content-Signal: search=yes,ai-train=no,use=reference
31+ Allow: /
32+
33+ User-agent: Amazonbot
34+ Disallow: /
35+
36+ User-agent: Applebot-Extended
37+ Disallow: /
38+
39+ User-agent: Bytespider
40+ Disallow: /
41+
42+ User-agent: CCBot
43+ Disallow: /
44+
45+ User-agent: ClaudeBot
46+ Disallow: /
47+
48+ User-agent: CloudflareBrowserRenderingCrawler
49+ Disallow: /
50+
51+ User-agent: Google-Extended
52+ Disallow: /
53+
54+ User-agent: GPTBot
55+ Disallow: /
56+
57+ User-agent: meta-externalagent
58+ Disallow: /
59+
60+ # END Cloudflare Managed Content
61+
162  user-agent: *
263 
364  disallow: *.ashx