NeuralCrawl

The Economist / robots.txt snapshot

← back to economist.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 1148 bytes · sha256 0d55cfa9c1f5b1da · raw

final URL: https://www.economist.com/robots.txt

1# robots.txt
2
3# Sitemaps
4
5Sitemap: https://www.economist.com/sitemap.xml
6Sitemap: https://www.economist.com/googlenews.xml
7
8# Bot Directives
9
10User-agent: GPTBot
11Disallow: /
12
13User-agent: Google-Extended
14Disallow: /
15
16User-agent: ChatGPT-User
17Disallow: /
18
19User-agent: CCBot
20Disallow: /
21
22User-agent: PiplBot
23Disallow: /
24
25User-agent: anthropic-ai
26Disallow: /
27
28User-agent: Claude-Web
29Disallow: /
30
31User-agent: TurnitinBot
32Disallow: /
33
34User-agent: PetalBot
35Disallow: /
36
37User-agent: MoodleBot
38Disallow: /
39
40User-agent: magpie-crawler
41Disallow: /
42
43User-agent: ia_archiver
44Disallow: /
45
46User-agent: Applebot-Extended
47Disallow: /
48
49User-agent: PerplexityBot
50Disallow: /
51
52User-agent: Perplexity-User
53Disallow: /
54
55User-Agent: Bytespider
56Disallow: /
57
58User-Agent: Mozilla/5.0 (compatible; parse.ly scraper/0.16; +http://parsely.com)
59Disallow: /checkout
60
61# Disallow all bots from select paths
62User-agent: *
63Disallow: /5605/
64Disallow: /pubads.g.doubleclick.net/
65Disallow: /assets/infographic/
66Disallow: /search/
67Disallow: /search?q=
68Disallow: /graphql
69Disallow: /subscribe/getstarted/
70Disallow: /checkout
71Disallow: /switch*
72Disallow: /audio-edition-podcast/*/index.xml