CNN / robots.txt
← back to cnn.com · change detected 1mo ago (2026-09-25T17:22:55Z) · modified +2 −4
What changed
- Formatting/comment-only change (no semantic difference)
Diff old (2026-09-24T23:05:48Z) → new (2026-09-25T17:22:55Z)
| @@ -76,6 +76,8 @@ | ||
| 76 | 76 | User-agent: SemrushBot-OCOB |
| 77 | 77 | User-agent: SemrushBot-SWA |
| 78 | 78 | User-agent: SeznamHomepageCrawler |
| 79 | + User-agent: ShapBot | |
| 80 | + User-agent: Shap-User | |
| 79 | 81 | User-agent: Sidetrade indexer bot |
| 80 | 82 | User-agent: TaraGroup Intelligence Bot |
| 81 | 83 | User-agent: Timpibot |
| @@ -127,7 +129,3 @@ | ||
| 127 | 129 | Disallow: /webview/ |
| 128 | 130 | User-agent: Googlebot-News |
| 129 | 131 | Disallow: /sponsor |
| 130 | - User-agent: ShapBot | |
| 131 | - Disallow: / | |
| 132 | - User-agent: Shap-User | |
| 133 | - Disallow: / |