CNN / robots.txt
← back to cnn.com · change detected 17d ago (2026-10-21T08:42:05Z) · modified +5 −0
What changed
- Blocked user-agent ‘Diffbot-User’ entirely
- Blocked user-agent ‘Firecrawl’ entirely
- Blocked user-agent ‘FirecrawlAgent’ entirely
- Blocked user-agent ‘NimbleCrawler’ entirely
- Blocked user-agent ‘TavilyBot’ entirely
Diff old (2026-10-15T00:17:21Z) → new (2026-10-21T08:42:05Z)
| @@ -89,6 +89,11 @@ | ||
| 89 | 89 | User-agent: YandexAdditional |
| 90 | 90 | User-agent: YandexAdditionalBot |
| 91 | 91 | User-agent: YouBot |
| 92 | + User-agent: FirecrawlAgent | |
| 93 | + User-agent: Firecrawl | |
| 94 | + User-agent: Diffbot-User | |
| 95 | + User-agent: TavilyBot | |
| 96 | + User-agent: NimbleCrawler | |
| 92 | 97 | Disallow: / |
| 93 | 98 | User-agent: * |
| 94 | 99 | Allow: /partners/ipad/live-video.json |