The Economist / robots.txt
← back to economist.com · change detected 3d ago (2026-08-20T18:05:43Z) · modified +16 −0
What changed
- 🤖 AIAdded Allow: /syndication$ for ChatGPT-User
- 🤖 AIAdded Allow: /syndication/ for ChatGPT-User
- 🤖 AIAdded Allow: /syndication$ for Claude-SearchBot
- 🤖 AIAdded Allow: /syndication/ for Claude-SearchBot
- 🤖 AIAdded Allow: /syndication$ for Claude-User
- 🤖 AIAdded Allow: /syndication/ for Claude-User
- 🤖 AIAdded Allow: /syndication$ for Claude-Web
- 🤖 AIAdded Allow: /syndication/ for Claude-Web
- 🤖 AIAdded Allow: /syndication$ for Google-Extended
- 🤖 AIAdded Allow: /syndication/ for Google-Extended
- 🤖 AIAdded Allow: /syndication$ for OAI-SearchBot
- 🤖 AIAdded Allow: /syndication/ for OAI-SearchBot
- 🤖 AIAdded Allow: /syndication$ for Perplexity-User
- 🤖 AIAdded Allow: /syndication/ for Perplexity-User
- 🤖 AIAdded Allow: /syndication$ for PerplexityBot
- 🤖 AIAdded Allow: /syndication/ for PerplexityBot
Diff old (2026-08-06T13:18:41Z) → new (2026-08-20T18:05:43Z)
| @@ -13,31 +13,43 @@ | ||
| 13 | 13 | User-agent: ChatGPT-User |
| 14 | 14 | Allow: /pro$ |
| 15 | 15 | Allow: /pro/ |
| 16 | + Allow: /syndication$ | |
| 17 | + Allow: /syndication/ | |
| 16 | 18 | Disallow: / |
| 17 | 19 | |
| 18 | 20 | User-agent: Claude-SearchBot |
| 19 | 21 | Allow: /pro$ |
| 20 | 22 | Allow: /pro/ |
| 23 | + Allow: /syndication$ | |
| 24 | + Allow: /syndication/ | |
| 21 | 25 | Disallow: / |
| 22 | 26 | |
| 23 | 27 | User-agent: Claude-User |
| 24 | 28 | Allow: /pro$ |
| 25 | 29 | Allow: /pro/ |
| 30 | + Allow: /syndication$ | |
| 31 | + Allow: /syndication/ | |
| 26 | 32 | Disallow: / |
| 27 | 33 | |
| 28 | 34 | User-agent: Claude-Web |
| 29 | 35 | Allow: /pro$ |
| 30 | 36 | Allow: /pro/ |
| 37 | + Allow: /syndication$ | |
| 38 | + Allow: /syndication/ | |
| 31 | 39 | Disallow: / |
| 32 | 40 | |
| 33 | 41 | User-agent: PerplexityBot |
| 34 | 42 | Allow: /pro$ |
| 35 | 43 | Allow: /pro/ |
| 44 | + Allow: /syndication$ | |
| 45 | + Allow: /syndication/ | |
| 36 | 46 | Disallow: / |
| 37 | 47 | |
| 38 | 48 | User-agent: Perplexity-User |
| 39 | 49 | Allow: /pro$ |
| 40 | 50 | Allow: /pro/ |
| 51 | + Allow: /syndication$ | |
| 52 | + Allow: /syndication/ | |
| 41 | 53 | Disallow: / |
| 42 | 54 | |
| 43 | 55 | # I have added OAI-SearchBot (OpenAI's live search agent) |
| @@ -45,6 +57,8 @@ | ||
| 45 | 57 | User-agent: OAI-SearchBot |
| 46 | 58 | Allow: /pro$ |
| 47 | 59 | Allow: /pro/ |
| 60 | + Allow: /syndication$ | |
| 61 | + Allow: /syndication/ | |
| 48 | 62 | Disallow: / |
| 49 | 63 | |
| 50 | 64 | # ========================================== |
| @@ -60,6 +74,8 @@ | ||
| 60 | 74 | User-agent: Google-Extended |
| 61 | 75 | Allow: /pro$ |
| 62 | 76 | Allow: /pro/ |
| 77 | + Allow: /syndication$ | |
| 78 | + Allow: /syndication/ | |
| 63 | 79 | Disallow: / |
| 64 | 80 | |
| 65 | 81 | User-agent: anthropic-ai |