NeuralCrawl

The Economist / robots.txt snapshot

← back to economist.com · fetched 2026-08-06T13:18:41Z (18d ago) · HTTP 200 · 2314 bytes · sha256 9ae18e3878ef0973 · raw

final URL: https://www.economist.com/robots.txt

1# robots.txt
2
3# Sitemaps
4
5Sitemap: https://www.economist.com/sitemap.xml
6Sitemap: https://www.economist.com/googlenews.xml
7
8# ==========================================
9# USER-FACING LLM BOTS
10# Allowed to fetch on behalf of a user
11# ==========================================
12
13User-agent: ChatGPT-User
14Allow: /pro$
15Allow: /pro/
16Disallow: /
17
18User-agent: Claude-SearchBot
19Allow: /pro$
20Allow: /pro/
21Disallow: /
22
23User-agent: Claude-User
24Allow: /pro$
25Allow: /pro/
26Disallow: /
27
28User-agent: Claude-Web
29Allow: /pro$
30Allow: /pro/
31Disallow: /
32
33User-agent: PerplexityBot
34Allow: /pro$
35Allow: /pro/
36Disallow: /
37
38User-agent: Perplexity-User
39Allow: /pro$
40Allow: /pro/
41Disallow: /
42
43# I have added OAI-SearchBot (OpenAI's live search agent)
44# as it behaves the same way as ChatGPT-User.
45User-agent: OAI-SearchBot
46Allow: /pro$
47Allow: /pro/
48Disallow: /
49
50# ==========================================
51# TRAINING BOTS & MASS SCRAPERS
52# Completely blocked from the entire site
53# ==========================================
54
55User-agent: GPTBot
56Disallow: /
57
58# Google-Extended is used by the Gemini app for searches on
59# behalf of the user but it also used for training
60User-agent: Google-Extended
61Allow: /pro$
62Allow: /pro/
63Disallow: /
64
65User-agent: anthropic-ai
66Disallow: /
67
68User-agent: Applebot-Extended
69Disallow: /
70
71User-Agent: Bytespider
72Disallow: /
73
74User-agent: CCBot
75Disallow: /
76
77User-agent: ClaudeBot
78Disallow: /
79
80# ==========================================
81# OTHER MISCELLANEOUS BOTS
82# Completely blocked
83# ==========================================
84
85User-agent: PiplBot
86Disallow: /
87
88User-agent: TurnitinBot
89Disallow: /
90
91User-agent: PetalBot
92Disallow: /
93
94User-agent: MoodleBot
95Disallow: /
96
97User-agent: magpie-crawler
98Disallow: /
99
100User-agent: ia_archiver
101Disallow: /
102
103User-Agent: Mozilla/5.0 (compatible; parse.ly scraper/0.16; +http://parsely.com)
104Disallow: /checkout
105
106# ==========================================
107# GENERAL CRAWLERS (e.g., Googlebot, Bingbot)
108# Allowed by default, but blocked from select paths
109# ==========================================
110User-agent: *
111Disallow: /5605/
112Disallow: /pubads.g.doubleclick.net/
113Disallow: /assets/infographic/
114Disallow: /search/
115Disallow: /search?q=
116Disallow: /graphql
117Disallow: /subscribe/getstarted/
118Disallow: /checkout
119Disallow: /switch*
120Disallow: /audio-edition-podcast/*/index.xml