NBC News / robots.txt
← back to nbcnews.com · change detected 15d ago (2026-10-22T21:05:52Z) · modified +9 −8
What changed
- 🤖 AIAdded Allow: /information/nbc-news-info/about-nbc-news-digital-n1232178 for Meta-WebIndexer
- 🤖 AIAdded Allow: /information/nbc-news-info/closed-captioning-n1307063 for Meta-WebIndexer
- 🤖 AIAdded Allow: /information/nbc-news-info/contact-us-n1232521 for Meta-WebIndexer
- 🤖 AIAdded Allow: /news/us-news/nbc-news-digital-editors-n893846 for Meta-WebIndexer
- 🤖 AIAdded Allow: /select for Meta-WebIndexer
- 🤖 AIAdded Allow: /select/ for Meta-WebIndexer
- 🤖 AIAdded Allow: /subscribe/supplemental-terms for Meta-WebIndexer
- 🤖 AIAdded Allow: /tips for Meta-WebIndexer
- 🤖 AIAdded Allow: /tips/ for Meta-WebIndexer
- 🤖 AIRemoved Allow: /about/today-rcna226078 for Meta-WebIndexer
- 🤖 AIRemoved Allow: /about/today/today-digital-masthead-rcna155416 for Meta-WebIndexer
- 🤖 AIRemoved Allow: /information/-us-send-today-your-questions-comments-t215381 for Meta-WebIndexer
- 🤖 AIRemoved Allow: /information/closed-captioning-t215383 for Meta-WebIndexer
- 🤖 AIRemoved Allow: /information/join-today-live-plaza-t250868 for Meta-WebIndexer
- 🤖 AIRemoved Allow: /shop for Meta-WebIndexer
- 🤖 AIRemoved Allow: /shop/ for Meta-WebIndexer
- 🤖 AIRemoved Allow: /start-today/app/supplemental-terms for Meta-WebIndexer
- Added Allow: /information/nbc-news-info/about-nbc-news-digital-n1232178 for meta-webindexer/1.1
- Added Allow: /information/nbc-news-info/closed-captioning-n1307063 for meta-webindexer/1.1
- Added Allow: /information/nbc-news-info/contact-us-n1232521 for meta-webindexer/1.1
- Added Allow: /news/us-news/nbc-news-digital-editors-n893846 for meta-webindexer/1.1
- Added Allow: /select for meta-webindexer/1.1
- Added Allow: /select/ for meta-webindexer/1.1
- Added Allow: /subscribe/supplemental-terms for meta-webindexer/1.1
- Added Allow: /tips for meta-webindexer/1.1
- Added Allow: /tips/ for meta-webindexer/1.1
- Removed Allow: /about/today-rcna226078 for meta-webindexer/1.1
- Removed Allow: /about/today/today-digital-masthead-rcna155416 for meta-webindexer/1.1
- Removed Allow: /information/-us-send-today-your-questions-comments-t215381 for meta-webindexer/1.1
- Removed Allow: /information/closed-captioning-t215383 for meta-webindexer/1.1
- Removed Allow: /information/join-today-live-plaza-t250868 for meta-webindexer/1.1
- Removed Allow: /shop for meta-webindexer/1.1
- Removed Allow: /shop/ for meta-webindexer/1.1
- Removed Allow: /start-today/app/supplemental-terms for meta-webindexer/1.1
Diff old (2026-10-22T02:51:55Z) → new (2026-10-22T21:05:52Z)
| @@ -295,14 +295,15 @@ | ||
| 295 | 295 | User-agent: Meta-WebIndexer |
| 296 | 296 | User-agent: meta-webindexer/1.1 |
| 297 | 297 | Disallow: / |
| 298 | - Allow: /shop | |
| 299 | - Allow: /shop/ | |
| 300 | - Allow: /about/today/today-digital-masthead-rcna155416 | |
| 301 | - Allow: /about/today-rcna226078 | |
| 302 | - Allow: /information/join-today-live-plaza-t250868 | |
| 303 | - Allow: /start-today/app/supplemental-terms | |
| 304 | - Allow: /information/closed-captioning-t215383 | |
| 305 | - Allow: /information/-us-send-today-your-questions-comments-t215381 | |
| 298 | + Allow: /select | |
| 299 | + Allow: /select/ | |
| 300 | + Allow: /information/nbc-news-info/about-nbc-news-digital-n1232178 | |
| 301 | + Allow: /subscribe/supplemental-terms | |
| 302 | + Allow: /information/nbc-news-info/contact-us-n1232521 | |
| 303 | + Allow: /news/us-news/nbc-news-digital-editors-n893846 | |
| 304 | + Allow: /tips | |
| 305 | + Allow: /tips/ | |
| 306 | + Allow: /information/nbc-news-info/closed-captioning-n1307063 | |
| 306 | 307 | |
| 307 | 308 | User-agent: MistralAI-User |
| 308 | 309 | Disallow: / |