Slate / robots.txt
← back to slate.com · change detected 20d ago (2026-07-17T18:17:47Z) · modified +10 −0
What changed
- Blocked user-agent ‘bedrockbot’ entirely
- Blocked user-agent ‘DeepSeekBot’ entirely
- Blocked user-agent ‘Doubaobot’ entirely
- Blocked user-agent ‘ERNIEBot’ entirely
- Blocked user-agent ‘KimiCrawler’ entirely
- Blocked user-agent ‘micro-crawl’ entirely
- Blocked user-agent ‘MoonshotBot’ entirely
- Blocked user-agent ‘MoonSpider’ entirely
- Blocked user-agent ‘QwenBot’ entirely
- Blocked user-agent ‘SalamandraVLM’ entirely
Diff old (2026-07-08T20:50:19Z) → new (2026-07-17T18:17:47Z)
| @@ -18,6 +18,7 @@ | ||
| 18 | 18 | User-agent: Ai2Bot-Dolma |
| 19 | 19 | User-agent: Amazonbot |
| 20 | 20 | User-agent: Applebot-Extended |
| 21 | + User-agent: bedrockbot | |
| 21 | 22 | User-agent: Bytespider |
| 22 | 23 | User-agent: CCBot |
| 23 | 24 | User-agent: ChatGLM-Spider |
| @@ -29,6 +30,9 @@ | ||
| 29 | 30 | User-agent: Dataprovider.com |
| 30 | 31 | User-agent: Datenbank Crawler |
| 31 | 32 | User-agent: dcrawl |
| 33 | + User-agent: DeepSeekBot | |
| 34 | + User-agent: Doubaobot | |
| 35 | + User-agent: ERNIEBot | |
| 32 | 36 | User-agent: FacebookBot |
| 33 | 37 | User-agent: Google-Extended |
| 34 | 38 | User-agent: GoogleOther |
| @@ -41,13 +45,17 @@ | ||
| 41 | 45 | User-agent: IndeedJobBot |
| 42 | 46 | User-agent: Kangaroo Bot |
| 43 | 47 | User-agent: KimiBot |
| 48 | + User-agent: KimiCrawler | |
| 44 | 49 | User-agent: KrawlerBot |
| 45 | 50 | User-agent: laion-huggingface-processor |
| 46 | 51 | User-agent: LCC |
| 47 | 52 | User-agent: meta-externalagent |
| 48 | 53 | User-agent: MetaInspector |
| 49 | 54 | User-agent: MetaJobBot |
| 55 | + User-agent: micro-crawl | |
| 50 | 56 | User-agent: MistralBot |
| 57 | + User-agent: MoonshotBot | |
| 58 | + User-agent: MoonSpider | |
| 51 | 59 | User-agent: netEstate Imprint Crawler |
| 52 | 60 | User-agent: newspaper |
| 53 | 61 | User-agent: Nutch |
| @@ -57,6 +65,8 @@ | ||
| 57 | 65 | User-agent: OpenindexSpider |
| 58 | 66 | User-agent: PanguBot |
| 59 | 67 | User-agent: Potions |
| 68 | + User-agent: QwenBot | |
| 69 | + User-agent: SalamandraVLM | |
| 60 | 70 | User-agent: SBIntuitionsBot |
| 61 | 71 | User-agent: Scrapy |
| 62 | 72 | User-agent: ServerHunterSpider |