NBC News / robots.txt
← back to nbcnews.com · change detected 16d ago (2026-10-22T02:51:55Z) · modified +25 −0
What changed
- 🤖 AIBlocked meta-externalfetcher (Meta) entirely
- 🤖 AIBlocked Meta-WebIndexer (Meta) entirely
- Blocked user-agent ‘meta-webindexer/1.1’ entirely
Diff old (2026-10-20T02:14:13Z) → new (2026-10-22T02:51:55Z)
| @@ -279,6 +279,31 @@ | ||
| 279 | 279 | Allow: /tips/ |
| 280 | 280 | Allow: /information/nbc-news-info/closed-captioning-n1307063 |
| 281 | 281 | |
| 282 | + User-agent: Meta-ExternalFetcher | |
| 283 | + User-agent: meta-externalfetcher | |
| 284 | + Disallow: / | |
| 285 | + Allow: /select | |
| 286 | + Allow: /select/ | |
| 287 | + Allow: /information/nbc-news-info/about-nbc-news-digital-n1232178 | |
| 288 | + Allow: /subscribe/supplemental-terms | |
| 289 | + Allow: /information/nbc-news-info/contact-us-n1232521 | |
| 290 | + Allow: /news/us-news/nbc-news-digital-editors-n893846 | |
| 291 | + Allow: /tips | |
| 292 | + Allow: /tips/ | |
| 293 | + Allow: /information/nbc-news-info/closed-captioning-n1307063 | |
| 294 | + | |
| 295 | + User-agent: Meta-WebIndexer | |
| 296 | + User-agent: meta-webindexer/1.1 | |
| 297 | + Disallow: / | |
| 298 | + Allow: /shop | |
| 299 | + Allow: /shop/ | |
| 300 | + Allow: /about/today/today-digital-masthead-rcna155416 | |
| 301 | + Allow: /about/today-rcna226078 | |
| 302 | + Allow: /information/join-today-live-plaza-t250868 | |
| 303 | + Allow: /start-today/app/supplemental-terms | |
| 304 | + Allow: /information/closed-captioning-t215383 | |
| 305 | + Allow: /information/-us-send-today-your-questions-comments-t215381 | |
| 306 | + | |
| 282 | 307 | User-agent: MistralAI-User |
| 283 | 308 | Disallow: / |
| 284 | 309 | Allow: /select |