X (Twitter) / robots.txt
← back to x.com · change detected 7d ago (2026-10-31T06:07:55Z) · modified +7 −0
What changed
- Blocked user-agent ‘__usage_ai-training’ entirely
Diff old (2026-10-30T11:48:22Z) → new (2026-10-31T06:07:55Z)
| @@ -95,6 +95,13 @@ | ||
| 95 | 95 | User-agent: meta-externalfetcher |
| 96 | 96 | Disallow: * |
| 97 | 97 | |
| 98 | + # Shared AI training preference (IETF draft-kelley-aipref-training-preference-signal), | |
| 99 | + # honored by Microsoft, Google, OpenAI, and Anthropic. This is a usage policy | |
| 100 | + # token, not a crawler: it does not change crawl access, and `User-agent: *` | |
| 101 | + # never matches it. | |
| 102 | + User-agent: __usage_ai-training | |
| 103 | + Disallow: / | |
| 104 | + | |
| 98 | 105 | # Every bot that might possibly read and respect this file |
| 99 | 106 | # ======================================================== |
| 100 | 107 | User-agent: * |