Slate / robots.txt
← back to slate.com · change detected 1d ago (2026-09-17T21:12:52Z) · modified +2 −1
What changed
- Added rules for user-agent ‘Dataprovider’
- Unblocked user-agent ‘Dataprovider.com’
- Added Disallow: /_ for Dataprovider.com
- Added Disallow: /comments/ for Dataprovider.com
- Added Disallow: /css for Dataprovider.com
- Added Disallow: /fonts for Dataprovider.com
- Added Disallow: /media for Dataprovider.com
- Added Disallow: /piano for Dataprovider.com
- Added Disallow: /search for Dataprovider.com
- Added Disallow: /static for Dataprovider.com
- Crawl-delay for Dataprovider.com: none → 10
Diff old (2026-09-09T00:14:16Z) → new (2026-09-17T21:12:52Z)
| @@ -33,7 +33,6 @@ | ||
| 33 | 33 | User-agent: Cotoyogi |
| 34 | 34 | User-agent: CragCrawler |
| 35 | 35 | User-agent: Crawlspace |
| 36 | - User-agent: Dataprovider.com | |
| 37 | 36 | User-agent: Datenbank Crawler |
| 38 | 37 | User-agent: dcrawl |
| 39 | 38 | User-agent: DeepSeekBot |
| @@ -170,6 +169,8 @@ | ||
| 170 | 169 | User-agent: DataForSEO |
| 171 | 170 | User-agent: DataForSEO Bot |
| 172 | 171 | User-agent: DataForSeoBot |
| 172 | + User-agent: Dataprovider | |
| 173 | + User-agent: Dataprovider.com | |
| 173 | 174 | User-agent: deepcrawl |
| 174 | 175 | User-agent: deepcrawl.com |
| 175 | 176 | User-agent: DomainStatsBot |