ABC News / robots.txt
← back to abcnews.go.com · change detected 1mo ago (2026-08-17T17:05:52Z) · modified +7 −4
What changed
- Blocked user-agent ‘Googlebot-News’ entirely
- Added sitemap https://abcnews.go.com/xmap
- Added sitemap https://abcnews.go.com/xmlLatestStories
- Added sitemap https://abcnews.go.com/xmlLatestVideos
- Removed sitemap https://abcnews.com/xmap
- Removed sitemap https://abcnews.com/xmlLatestStories
- Removed sitemap https://abcnews.com/xmlLatestVideos
Diff old (2026-06-20T01:10:30Z) → new (2026-08-17T17:05:52Z)
| @@ -1,10 +1,10 @@ | ||
| 1 | 1 | |
| 2 | - # robots.txt for https://abcnews.com/ | |
| 2 | + # robots.txt for https://abcnews.go.com/ | |
| 3 | 3 | # |
| 4 | 4 | |
| 5 | - Sitemap: https://abcnews.com/xmap | |
| 6 | - Sitemap: https://abcnews.com/xmlLatestStories | |
| 7 | - Sitemap: https://abcnews.com/xmlLatestVideos | |
| 5 | + Sitemap: https://abcnews.go.com/xmap | |
| 6 | + Sitemap: https://abcnews.go.com/xmlLatestStories | |
| 7 | + Sitemap: https://abcnews.go.com/xmlLatestVideos | |
| 8 | 8 | |
| 9 | 9 | User-agent: claritybot |
| 10 | 10 | Disallow: / |
| @@ -34,6 +34,9 @@ | ||
| 34 | 34 | Disallow: / |
| 35 | 35 | |
| 36 | 36 | User-agent: Bytespider |
| 37 | + Disallow: / | |
| 38 | + | |
| 39 | + User-agent: Googlebot-News | |
| 37 | 40 | Disallow: / |
| 38 | 41 | |
| 39 | 42 | User-agent: * |