The New York Times / robots.txt
← back to nytimes.com · change detected 25d ago (2026-08-24T19:21:08Z) · modified +15 −0
What changed
- Blocked user-agent ‘Brightbot’ entirely
- Blocked user-agent ‘Diffbot-User’ entirely
- Blocked user-agent ‘ExaBot’ entirely
- Blocked user-agent ‘FirecrawlAgent’ entirely
- Blocked user-agent ‘ShapBot’ entirely
Diff old (2026-08-12T21:29:54Z) → new (2026-08-24T19:21:08Z)
| @@ -162,6 +162,9 @@ | ||
| 162 | 162 | User-agent: BLEXBot |
| 163 | 163 | Disallow: / |
| 164 | 164 | |
| 165 | + User-agent: Brightbot | |
| 166 | + Disallow: / | |
| 167 | + | |
| 165 | 168 | User-agent: Bytespider |
| 166 | 169 | Disallow: / |
| 167 | 170 | |
| @@ -192,13 +195,22 @@ | ||
| 192 | 195 | User-agent: Diffbot |
| 193 | 196 | Disallow: / |
| 194 | 197 | |
| 198 | + User-agent: Diffbot-User | |
| 199 | + Disallow: / | |
| 200 | + | |
| 195 | 201 | User-agent: DuckAssistBot |
| 196 | 202 | Disallow: / |
| 197 | 203 | |
| 198 | 204 | User-agent: EchoboxBot |
| 199 | 205 | Disallow: / |
| 200 | 206 | |
| 207 | + User-agent: ExaBot | |
| 208 | + Disallow: / | |
| 209 | + | |
| 201 | 210 | User-agent: FacebookBot |
| 211 | + Disallow: / | |
| 212 | + | |
| 213 | + User-agent: FirecrawlAgent | |
| 202 | 214 | Disallow: / |
| 203 | 215 | |
| 204 | 216 | User-agent: FriendlyCrawler |
| @@ -282,6 +294,9 @@ | ||
| 282 | 294 | User-agent: SeznamHomepageCrawler |
| 283 | 295 | Disallow: / |
| 284 | 296 | |
| 297 | + User-agent: ShapBot | |
| 298 | + Disallow: / | |
| 299 | + | |
| 285 | 300 | User-agent: TaraGroup Intelligent Bot |
| 286 | 301 | Disallow: / |
| 287 | 302 |