Slate / robots.txt
← back to slate.com · change detected 2mo ago (2026-08-25T19:41:43Z) · modified +5 −0
What changed
- Added rules for user-agent ‘captify-crawler’
- Added rules for user-agent ‘GnowitNewsbot’
- Added rules for user-agent ‘peer39_crawler’
- Blocked user-agent ‘Reflectionbot’ entirely
- Added rules for user-agent ‘SirdataBot’
Diff old (2026-07-28T22:23:38Z) → new (2026-08-25T19:41:43Z)
| @@ -66,6 +66,7 @@ | ||
| 66 | 66 | User-agent: PanguBot |
| 67 | 67 | User-agent: Potions |
| 68 | 68 | User-agent: QwenBot |
| 69 | + User-agent: Reflectionbot | |
| 69 | 70 | User-agent: SalamandraVLM |
| 70 | 71 | User-agent: SBIntuitionsBot |
| 71 | 72 | User-agent: Scrapy |
| @@ -123,6 +124,7 @@ | ||
| 123 | 124 | User-agent: Buck |
| 124 | 125 | User-agent: CaliberBot |
| 125 | 126 | User-agent: Caliperbot |
| 127 | + User-agent: captify-crawler | |
| 126 | 128 | User-agent: channable |
| 127 | 129 | User-agent: CheckMarkNetwork |
| 128 | 130 | User-agent: Cincraw |
| @@ -168,6 +170,7 @@ | ||
| 168 | 170 | User-agent: factset_spyderbot |
| 169 | 171 | User-agent: FDL Stats Bot |
| 170 | 172 | User-agent: Fedicabot |
| 173 | + User-agent: GnowitNewsbot | |
| 171 | 174 | User-agent: Google AdsBot |
| 172 | 175 | User-agent: Google AdSense |
| 173 | 176 | User-agent: Google Digital Asset Links |
| @@ -254,6 +257,7 @@ | ||
| 254 | 257 | User-agent: panscient.com |
| 255 | 258 | User-agent: parse.ly scraper |
| 256 | 259 | User-agent: PartnerOptimizer-bot |
| 260 | + User-agent: peer39_crawler | |
| 257 | 261 | User-agent: Prerender |
| 258 | 262 | User-agent: Pro Sitemaps |
| 259 | 263 | User-agent: Pro Sitemaps Generator |
| @@ -296,6 +300,7 @@ | ||
| 296 | 300 | User-agent: SERankingBacklinksBot |
| 297 | 301 | User-agent: SerpMinerBot |
| 298 | 302 | User-agent: serpstatbot |
| 303 | + User-agent: SirdataBot | |
| 299 | 304 | User-agent: SiteAuditBot |
| 300 | 305 | User-agent: Sitebulb |
| 301 | 306 | User-agent: SiteCheck-sitecrawl |