Slate / robots.txt
← back to slate.com · change detected 23d ago (2026-08-26T20:03:19Z) · modified +18 −0
What changed
- Blocked user-agent ‘ApifyBot’ entirely
- Blocked user-agent ‘ApifyWebsiteContentCrawler’ entirely
- Blocked user-agent ‘Bravebot’ entirely
- Blocked user-agent ‘Brightbot’ entirely
- Blocked user-agent ‘CragCrawler’ entirely
- Blocked user-agent ‘Crawlspace’ entirely
- 🤖 AIBlocked Diffbot (Diffbot) entirely
- Blocked user-agent ‘ExaBot’ entirely
- Blocked user-agent ‘FirecrawlAgent’ entirely
- Blocked user-agent ‘HenkBot’ entirely
- Blocked user-agent ‘Mozilla-Tabstack’ entirely
- Blocked user-agent ‘Querit-SearchBot’ entirely
- Blocked user-agent ‘QueritBot’ entirely
- Blocked user-agent ‘ShapBot’ entirely
- Blocked user-agent ‘TavilyBot’ entirely
- Blocked user-agent ‘Terra Cotta’ entirely
- Blocked user-agent ‘TerraCotta’ entirely
- 🤖 AIBlocked YouBot (You.com) entirely
Diff old (2026-08-25T19:41:43Z) → new (2026-08-26T20:03:19Z)
| @@ -17,8 +17,12 @@ | ||
| 17 | 17 | User-agent: AI2Bot |
| 18 | 18 | User-agent: Ai2Bot-Dolma |
| 19 | 19 | User-agent: Amazonbot |
| 20 | + User-agent: ApifyBot | |
| 21 | + User-agent: ApifyWebsiteContentCrawler | |
| 20 | 22 | User-agent: Applebot-Extended |
| 21 | 23 | User-agent: bedrockbot |
| 24 | + User-agent: Bravebot | |
| 25 | + User-agent: Brightbot | |
| 22 | 26 | User-agent: Bytespider |
| 23 | 27 | User-agent: CCBot |
| 24 | 28 | User-agent: ChatGLM-Spider |
| @@ -27,17 +31,23 @@ | ||
| 27 | 31 | User-agent: cohere-training-data-crawler |
| 28 | 32 | User-agent: CohereBot |
| 29 | 33 | User-agent: Cotoyogi |
| 34 | + User-agent: CragCrawler | |
| 35 | + User-agent: Crawlspace | |
| 30 | 36 | User-agent: Dataprovider.com |
| 31 | 37 | User-agent: Datenbank Crawler |
| 32 | 38 | User-agent: dcrawl |
| 33 | 39 | User-agent: DeepSeekBot |
| 40 | + User-agent: Diffbot | |
| 34 | 41 | User-agent: Doubaobot |
| 35 | 42 | User-agent: ERNIEBot |
| 43 | + User-agent: ExaBot | |
| 36 | 44 | User-agent: FacebookBot |
| 45 | + User-agent: FirecrawlAgent | |
| 37 | 46 | User-agent: Google-Extended |
| 38 | 47 | User-agent: GoogleOther |
| 39 | 48 | User-agent: GPTBot |
| 40 | 49 | User-agent: HelloworkJobPostingBot |
| 50 | + User-agent: HenkBot | |
| 41 | 51 | User-agent: HTTrack |
| 42 | 52 | User-agent: HTTrack 3.0 |
| 43 | 53 | User-agent: ICC-Crawler |
| @@ -56,6 +66,7 @@ | ||
| 56 | 66 | User-agent: MistralBot |
| 57 | 67 | User-agent: MoonshotBot |
| 58 | 68 | User-agent: MoonSpider |
| 69 | + User-agent: Mozilla-Tabstack | |
| 59 | 70 | User-agent: netEstate Imprint Crawler |
| 60 | 71 | User-agent: newspaper |
| 61 | 72 | User-agent: Nutch |
| @@ -65,18 +76,25 @@ | ||
| 65 | 76 | User-agent: OpenindexSpider |
| 66 | 77 | User-agent: PanguBot |
| 67 | 78 | User-agent: Potions |
| 79 | + User-agent: Querit-SearchBot | |
| 80 | + User-agent: QueritBot | |
| 68 | 81 | User-agent: QwenBot |
| 69 | 82 | User-agent: Reflectionbot |
| 70 | 83 | User-agent: SalamandraVLM |
| 71 | 84 | User-agent: SBIntuitionsBot |
| 72 | 85 | User-agent: Scrapy |
| 73 | 86 | User-agent: ServerHunterSpider |
| 87 | + User-agent: ShapBot | |
| 74 | 88 | User-agent: Spider |
| 75 | 89 | User-agent: StatsDroneBot |
| 90 | + User-agent: TavilyBot | |
| 91 | + User-agent: Terra Cotta | |
| 92 | + User-agent: TerraCotta | |
| 76 | 93 | User-agent: Thinkbot |
| 77 | 94 | User-agent: Timpibot |
| 78 | 95 | User-agent: VelenPublicWebCrawler |
| 79 | 96 | User-agent: webzio-extended |
| 97 | + User-agent: YouBot | |
| 80 | 98 | Disallow: / |
| 81 | 99 | |
| 82 | 100 | # Intelligence Gatherer, SEO Crawler |