Slate / robots.txt
← back to slate.com · change detected 2mo ago (2026-09-09T00:14:16Z) · modified +0 −1
What changed
- Removed all rules for user-agent ‘Prerender’
Diff old (2026-08-26T20:03:19Z) → new (2026-09-09T00:14:16Z)
| @@ -276,7 +276,6 @@ | ||
| 276 | 276 | User-agent: parse.ly scraper |
| 277 | 277 | User-agent: PartnerOptimizer-bot |
| 278 | 278 | User-agent: peer39_crawler |
| 279 | - User-agent: Prerender | |
| 280 | 279 | User-agent: Pro Sitemaps |
| 281 | 280 | User-agent: Pro Sitemaps Generator |
| 282 | 281 | User-agent: productsup.io/crawler |