NeuralCrawl

CNN / robots.txt snapshot

← back to cnn.com · fetched 2026-10-21T20:48:03Z (16d ago) · HTTP 200 · 3671 bytes · sha256 2f94c14d6285e519 · raw

final URL: https://www.cnn.com/robots.txt

1Sitemap: https://www.cnn.com/sitemap/news.xml
2Sitemap: https://www.cnn.com/sitemap/article/cnn-underscored.xml
3Sitemap: https://www.cnn.com/sitemap/section/cnn-underscored.xml
4Sitemap: https://www.cnn.com/sitemap/section/politics.xml
5Sitemap: https://www.cnn.com/sitemap/article/opinions.xml
6Sitemap: https://www.cnn.com/sitemap/article.xml
7Sitemap: https://www.cnn.com/sitemap/section.xml
8Sitemap: https://www.cnn.com/sitemap/video.xml
9Sitemap: https://www.cnn.com/sitemap/gallery.xml
10Sitemap: https://www.cnn.com/sitemap/markets/stocks.xml
11Sitemap: https://www.cnn.com/sitemap/live-story.xml
12Sitemap: https://www.cnn.com/sitemap/election-center/politics.xml
13Sitemap: https://www.cnn.com/sitemap/tve.xml
14User-agent: AI2Bot
15User-agent: Ai2Bot-Dolma
16User-agent: AliyunSecBot
17User-agent: Amazonbot
18User-agent: amzn-searchbot
19User-agent: amzn-user
20User-agent: anthropic-ai
21User-agent: Applebot-Extended
22User-agent: Archive.org_bot
23User-agent: AwarioRssBot
24User-agent: AwarioSmartBot
25User-agent: Brightbot 1.0
26User-agent: Bytespider
27User-agent: CCBot
28User-agent: ChatGPT-User
29User-agent: ClaudeBot
30User-agent: Claude-SearchBot
31User-agent: Claude-User
32User-agent: Claude-Web
33User-agent: cohere-ai
34User-agent: cohere-training-data-crawler
35User-agent: Crawlspace
36User-agent: DataForSeoBot
37User-agent: Diffbot
38User-agent: Diffbot-User
39User-agent: DuckAssistBot
40User-agent: EchoboxBot
41User-agent: Exabot
42User-agent: ExaSearchBot
43User-agent: ExaUser
44User-agent: FacebookBot
45User-agent: Firecrawl
46User-agent: FirecrawlAgent
47User-agent: FriendlyCrawler
48User-agent: GPTBot
49User-agent: GoogleOther
50User-agent: GoogleOther-Image
51User-agent: GoogleOther-Video
52User-Agent: IAB-Tech-Lab
53User-agent: iaskspider/2.0
54User-agent: ICC-Crawler
55User-agent: img2dataset
56User-agent: ISSCyberRiskCrawler
57User-agent: ImagesiftBot
58User-agent: Kangaroo Bot
59User-agent: magpie-crawler
60User-agent: MistralAI-user
61User-agent: MyCentralAIScraperBot
62User-agent: NewsNow
63User-agent: news-please
64User-agent: NimbleCrawler
65User-agent: OAI-SearchBot
66User-agent: omgili
67User-agent: omgilibot
68User-agent: PanguBot
69User-agent: Panscient
70User-agent: PiplBot
71User-agent: PerplexityBot
72User-agent: Perplexity-User
73User-agent: PetalBot
74User-agent: Poseidon Research Crawler
75User-agent: QuillBot
76User-agent: quillbot.com
77User-agent: Quora-Bot
78User-agent: SBIntuitionsBot
79User-agent: Scrapy
80User-agent: SeekrBot
81User-agent: SemrushBot-OCOB
82User-agent: SemrushBot-SWA
83User-agent: SeznamHomepageCrawler
84User-agent: ShapBot
85User-agent: Shap-User
86User-agent: Sidetrade indexer bot
87User-agent: TaraGroup Intelligence Bot
88User-agent: TavilyBot
89User-agent: Timpibot
90User-agent: TurnitinBot
91User-agent: VelenPublicWebCrawler
92User-agent: ViennaTinyBot
93User-agent: Webzio-Extended
94User-agent: YandexAdditional
95User-agent: YandexAdditionalBot
96User-agent: YouBot
97Disallow: /
98User-agent: *
99Allow: /partners/ipad/live-video.json
100Disallow: /*.jsx$
101Disallow: *.jsx$
102Disallow: /*.jsx/
103Disallow: *.jsx?
104Disallow: /ads/
105Disallow: /aol/
106Disallow: /api/
107Disallow: /beta/
108Disallow: /browsers/
109Disallow: /cl/
110Disallow: /cnews/
111Disallow: /cnn_adspaces
112Disallow: /cnnbeta/
113Disallow: /cnnintl_adspaces
114Disallow: /development
115Disallow: /editionssi
116Disallow: /help/cnnx.html
117Disallow: /media/sites/js/bundles
118Disallow: /NewsPass
119Disallow: /NOKIA
120Disallow: /partners/
121Disallow: /pipeline/
122Disallow: /pointroll/
123Disallow: /POLLSERVER/
124Disallow: /pr/
125Disallow: /PV/
126Disallow: /Quickcast/
127Disallow: /quickcast/
128Disallow: /QUICKNEWS/
129Disallow: /search
130Disallow: /subscriptions/video-docs
131Disallow: /terms
132Disallow: /test/
133Disallow: /virtual/
134Disallow: /WEB-INF/
135Disallow: /web.projects/
136Disallow: /webview/
137User-agent: Googlebot-News
138Disallow: /sponsor