NeuralCrawl

The Intercept / robots.txt snapshot

← back to theintercept.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 518 bytes · sha256 4b9315431c7be783 · raw

final URL: https://theintercept.com/robots.txt

1# Start Block AI Crawlers
2User-agent: Amazonbot
3Disallow: /
4
5User-agent: anthropic-ai
6Disallow: /
7
8User-agent: AhrefsBot
9Disallow: /
10
11User-Agent: PerplexityBot
12Disallow: /
13
14User-agent: CCBot
15Disallow: /
16
17User-agent: Google-Extended
18Disallow: /
19
20User-agent: GPTBot
21Disallow: /
22
23User-agent: ChatGPT-User
24Disallow: /
25# End Block AI Crawlers
26
27 # START YOAST BLOCK
28# ---------------------------
29User-agent: *
30Disallow:
31
32Sitemap: https://theintercept.com/sitemap_index.xml
33# ---------------------------
34# END YOAST BLOCK