The Economist / robots.txt
← back to economist.com · change detected 15d ago (2026-07-20T19:11:56Z) · modified +19 −1
What changed
- 🤖 AIBlocked Claude-SearchBot (Anthropic) entirely
- 🤖 AIBlocked Claude-User (Anthropic) entirely
- 🤖 AIBlocked ClaudeBot (Anthropic) entirely
- 🤖 AIAdded Allow: /pro$ for ChatGPT-User
- 🤖 AIAdded Allow: /pro$ for Claude-Web
- 🤖 AIAdded Allow: /pro$ for OAI-SearchBot
- 🤖 AIAdded Allow: /pro$ for Perplexity-User
- 🤖 AIAdded Allow: /pro$ for PerplexityBot
Diff old (2026-07-17T12:07:56Z) → new (2026-07-20T19:11:56Z)
| @@ -7,28 +7,43 @@ | ||
| 7 | 7 | |
| 8 | 8 | # ========================================== |
| 9 | 9 | # USER-FACING LLM BOTS |
| 10 | - # Allowed to fetch /pro/ on behalf of a user | |
| 10 | + # Allowed to fetch on behalf of a user | |
| 11 | 11 | # ========================================== |
| 12 | 12 | |
| 13 | 13 | User-agent: ChatGPT-User |
| 14 | + Allow: /pro$ | |
| 15 | + Allow: /pro/ | |
| 16 | + Disallow: / | |
| 17 | + | |
| 18 | + User-agent: Claude-SearchBot | |
| 19 | + Allow: /pro$ | |
| 20 | + Allow: /pro/ | |
| 21 | + Disallow: / | |
| 22 | + | |
| 23 | + User-agent: Claude-User | |
| 24 | + Allow: /pro$ | |
| 14 | 25 | Allow: /pro/ |
| 15 | 26 | Disallow: / |
| 16 | 27 | |
| 17 | 28 | User-agent: Claude-Web |
| 29 | + Allow: /pro$ | |
| 18 | 30 | Allow: /pro/ |
| 19 | 31 | Disallow: / |
| 20 | 32 | |
| 21 | 33 | User-agent: PerplexityBot |
| 34 | + Allow: /pro$ | |
| 22 | 35 | Allow: /pro/ |
| 23 | 36 | Disallow: / |
| 24 | 37 | |
| 25 | 38 | User-agent: Perplexity-User |
| 39 | + Allow: /pro$ | |
| 26 | 40 | Allow: /pro/ |
| 27 | 41 | Disallow: / |
| 28 | 42 | |
| 29 | 43 | # I have added OAI-SearchBot (OpenAI's live search agent) |
| 30 | 44 | # as it behaves the same way as ChatGPT-User. |
| 31 | 45 | User-agent: OAI-SearchBot |
| 46 | + Allow: /pro$ | |
| 32 | 47 | Allow: /pro/ |
| 33 | 48 | Disallow: / |
| 34 | 49 | |
| @@ -55,6 +70,9 @@ | ||
| 55 | 70 | Disallow: / |
| 56 | 71 | |
| 57 | 72 | User-agent: CCBot |
| 73 | + Disallow: / | |
| 74 | + | |
| 75 | + User-agent: ClaudeBot | |
| 58 | 76 | Disallow: / |
| 59 | 77 | |
| 60 | 78 | # ========================================== |