The Economist / robots.txt
← back to economist.com · change detected 18d ago (2026-07-17T12:07:56Z) · modified +49 −21
What changed
- 🤖 AIBlocked OAI-SearchBot (OpenAI) entirely
- 🤖 AIAdded Allow: /pro/ for ChatGPT-User
- 🤖 AIAdded Allow: /pro/ for Claude-Web
- 🤖 AIAdded Allow: /pro/ for Perplexity-User
- 🤖 AIAdded Allow: /pro/ for PerplexityBot
Diff old (2026-06-20T01:10:30Z) → new (2026-07-17T12:07:56Z)
| @@ -5,27 +5,64 @@ | ||
| 5 | 5 | Sitemap: https://www.economist.com/sitemap.xml |
| 6 | 6 | Sitemap: https://www.economist.com/googlenews.xml |
| 7 | 7 | |
| 8 | - # Bot Directives | |
| 8 | + # ========================================== | |
| 9 | + # USER-FACING LLM BOTS | |
| 10 | + # Allowed to fetch /pro/ on behalf of a user | |
| 11 | + # ========================================== | |
| 12 | + | |
| 13 | + User-agent: ChatGPT-User | |
| 14 | + Allow: /pro/ | |
| 15 | + Disallow: / | |
| 16 | + | |
| 17 | + User-agent: Claude-Web | |
| 18 | + Allow: /pro/ | |
| 19 | + Disallow: / | |
| 20 | + | |
| 21 | + User-agent: PerplexityBot | |
| 22 | + Allow: /pro/ | |
| 23 | + Disallow: / | |
| 24 | + | |
| 25 | + User-agent: Perplexity-User | |
| 26 | + Allow: /pro/ | |
| 27 | + Disallow: / | |
| 28 | + | |
| 29 | + # I have added OAI-SearchBot (OpenAI's live search agent) | |
| 30 | + # as it behaves the same way as ChatGPT-User. | |
| 31 | + User-agent: OAI-SearchBot | |
| 32 | + Allow: /pro/ | |
| 33 | + Disallow: / | |
| 34 | + | |
| 35 | + # ========================================== | |
| 36 | + # TRAINING BOTS & MASS SCRAPERS | |
| 37 | + # Completely blocked from the entire site | |
| 38 | + # ========================================== | |
| 9 | 39 | |
| 10 | 40 | User-agent: GPTBot |
| 11 | 41 | Disallow: / |
| 12 | 42 | |
| 43 | + # Google-Extended is used by the Gemini app for searches on | |
| 44 | + # behalf of the user but it also used for training | |
| 13 | 45 | User-agent: Google-Extended |
| 14 | 46 | Disallow: / |
| 15 | 47 | |
| 16 | - User-agent: ChatGPT-User | |
| 48 | + User-agent: anthropic-ai | |
| 49 | + Disallow: / | |
| 50 | + | |
| 51 | + User-agent: Applebot-Extended | |
| 52 | + Disallow: / | |
| 53 | + | |
| 54 | + User-Agent: Bytespider | |
| 17 | 55 | Disallow: / |
| 18 | 56 | |
| 19 | 57 | User-agent: CCBot |
| 20 | 58 | Disallow: / |
| 21 | 59 | |
| 60 | + # ========================================== | |
| 61 | + # OTHER MISCELLANEOUS BOTS | |
| 62 | + # Completely blocked | |
| 63 | + # ========================================== | |
| 64 | + | |
| 22 | 65 | User-agent: PiplBot |
| 23 | - Disallow: / | |
| 24 | - | |
| 25 | - User-agent: anthropic-ai | |
| 26 | - Disallow: / | |
| 27 | - | |
| 28 | - User-agent: Claude-Web | |
| 29 | 66 | Disallow: / |
| 30 | 67 | |
| 31 | 68 | User-agent: TurnitinBot |
| @@ -43,22 +80,13 @@ | ||
| 43 | 80 | User-agent: ia_archiver |
| 44 | 81 | Disallow: / |
| 45 | 82 | |
| 46 | - User-agent: Applebot-Extended | |
| 47 | - Disallow: / | |
| 48 | - | |
| 49 | - User-agent: PerplexityBot | |
| 50 | - Disallow: / | |
| 51 | - | |
| 52 | - User-agent: Perplexity-User | |
| 53 | - Disallow: / | |
| 54 | - | |
| 55 | - User-Agent: Bytespider | |
| 56 | - Disallow: / | |
| 57 | - | |
| 58 | 83 | User-Agent: Mozilla/5.0 (compatible; parse.ly scraper/0.16; +http://parsely.com) |
| 59 | 84 | Disallow: /checkout |
| 60 | 85 | |
| 61 | - # Disallow all bots from select paths | |
| 86 | + # ========================================== | |
| 87 | + # GENERAL CRAWLERS (e.g., Googlebot, Bingbot) | |
| 88 | + # Allowed by default, but blocked from select paths | |
| 89 | + # ========================================== | |
| 62 | 90 | User-agent: * |
| 63 | 91 | Disallow: /5605/ |
| 64 | 92 | Disallow: /pubads.g.doubleclick.net/ |