The New York Times / robots.txt
← back to nytimes.com · change detected 2mo ago (2026-08-27T20:19:03Z) · modified +3 −0
What changed
- Blocked user-agent ‘ExaSearchBot’ entirely
Diff old (2026-08-24T19:21:08Z) → new (2026-08-27T20:19:03Z)
| @@ -207,6 +207,9 @@ | ||
| 207 | 207 | User-agent: ExaBot |
| 208 | 208 | Disallow: / |
| 209 | 209 | |
| 210 | + User-agent: ExaSearchBot | |
| 211 | + Disallow: / | |
| 212 | + | |
| 210 | 213 | User-agent: FacebookBot |
| 211 | 214 | Disallow: / |
| 212 | 215 |