xAI / robots.txt
← back to x.ai · change detected 4d ago (2026-09-15T02:23:28Z) · modified +2 −13
What changed
- 🤖 AIRemoved rules for AI crawler Applebot-Extended (Apple)
- 🤖 AIRemoved rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AIRemoved rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AIRemoved rules for AI crawler Google-Extended (Google)
- 🤖 AIRemoved rules for AI crawler GPTBot (OpenAI)
- 🤖 AIRemoved rules for AI crawler PerplexityBot (Perplexity)
Diff old (2026-08-28T02:24:20Z) → new (2026-09-15T02:23:28Z)
| @@ -1,20 +1,9 @@ | ||
| 1 | 1 | User-agent: * |
| 2 | - Allow: / | |
| 3 | - Disallow: /tools/ | |
| 4 | - | |
| 5 | - User-agent: GPTBot | |
| 6 | - User-agent: ChatGPT-User | |
| 7 | - User-agent: PerplexityBot | |
| 8 | - User-agent: ClaudeBot | |
| 9 | - User-agent: Google-Extended | |
| 10 | - User-agent: Applebot-Extended | |
| 11 | 2 | Allow: / |
| 12 | 3 | Disallow: /tools/ |
| 13 | 4 | |
| 14 | 5 | Sitemap: https://x.ai/sitemap.xml |
| 15 | 6 | |
| 16 | - # Content Signals (draft) — declare AI/search usage preferences | |
| 7 | + # Content Signals — declare AI/search usage preferences | |
| 17 | 8 | # See: https://contentsignals.org/ |
| 18 | - Content-Signal: ai-train=no, search=yes, ai-input=no | |
| 19 | - # Guides are written to be cited; allow answer-engine input on this tree only. | |
| 20 | - Content-Signal: /bot/guides ai-train=no, search=yes, ai-input=yes | |
| 9 | + Content-Signal: ai-train=yes, search=yes, ai-input=yes |