CNN / robots.txt
← back to cnn.com · change detected 17d ago (2026-10-21T14:45:05Z) · modified +5 −5
What changed
- Formatting/comment-only change (no semantic difference)
Diff old (2026-10-21T08:42:05Z) → new (2026-10-21T14:45:05Z)
| @@ -35,12 +35,15 @@ | ||
| 35 | 35 | User-agent: Crawlspace |
| 36 | 36 | User-agent: DataForSeoBot |
| 37 | 37 | User-agent: Diffbot |
| 38 | + User-agent: Diffbot-User | |
| 38 | 39 | User-agent: DuckAssistBot |
| 39 | 40 | User-agent: EchoboxBot |
| 40 | 41 | User-agent: Exabot |
| 41 | 42 | User-agent: ExaSearchBot |
| 42 | 43 | User-agent: ExaUser |
| 43 | 44 | User-agent: FacebookBot |
| 45 | + User-agent: Firecrawl | |
| 46 | + User-agent: FirecrawlAgent | |
| 44 | 47 | User-agent: FriendlyCrawler |
| 45 | 48 | User-agent: GPTBot |
| 46 | 49 | User-agent: GoogleOther |
| @@ -58,6 +61,7 @@ | ||
| 58 | 61 | User-agent: MyCentralAIScraperBot |
| 59 | 62 | User-agent: NewsNow |
| 60 | 63 | User-agent: news-please |
| 64 | + User-agent: NimbleCrawler | |
| 61 | 65 | User-agent: OAI-SearchBot |
| 62 | 66 | User-agent: omgili |
| 63 | 67 | User-agent: omgilibot |
| @@ -81,6 +85,7 @@ | ||
| 81 | 85 | User-agent: Shap-User |
| 82 | 86 | User-agent: Sidetrade indexer bot |
| 83 | 87 | User-agent: TaraGroup Intelligence Bot |
| 88 | + User-agent: TavilyBot | |
| 84 | 89 | User-agent: Timpibot |
| 85 | 90 | User-agent: TurnitinBot |
| 86 | 91 | User-agent: VelenPublicWebCrawler |
| @@ -89,11 +94,6 @@ | ||
| 89 | 94 | User-agent: YandexAdditional |
| 90 | 95 | User-agent: YandexAdditionalBot |
| 91 | 96 | User-agent: YouBot |
| 92 | - User-agent: FirecrawlAgent | |
| 93 | - User-agent: Firecrawl | |
| 94 | - User-agent: Diffbot-User | |
| 95 | - User-agent: TavilyBot | |
| 96 | - User-agent: NimbleCrawler | |
| 97 | 97 | Disallow: / |
| 98 | 98 | User-agent: * |
| 99 | 99 | Allow: /partners/ipad/live-video.json |