NeuralCrawl

Slate / robots.txt

← back to slate.com · change detected 27d ago (2026-07-08T20:50:19Z) · modified +20 −0

What changed

Diff old (2026-06-20T01:10:30Z) → new (2026-07-08T20:50:19Z)

@@ -14,6 +14,7 @@
1414  # AI Data Scraper, Scraper
1515  User-agent: 008
1616  User-agent: AdsTxtCrawler
17+ User-agent: AI2Bot
1718  User-agent: Ai2Bot-Dolma
1819  User-agent: Amazonbot
1920  User-agent: Applebot-Extended
@@ -23,6 +24,7 @@
2324  User-agent: ClaudeBot
2425  User-agent: CloudVertexBot
2526  User-agent: cohere-training-data-crawler
27+ User-agent: CohereBot
2628  User-agent: Cotoyogi
2729  User-agent: Dataprovider.com
2830  User-agent: Datenbank Crawler
@@ -38,12 +40,14 @@
3840  User-agent: imageSpider
3941  User-agent: IndeedJobBot
4042  User-agent: Kangaroo Bot
43+ User-agent: KimiBot
4144  User-agent: KrawlerBot
4245  User-agent: laion-huggingface-processor
4346  User-agent: LCC
4447  User-agent: meta-externalagent
4548  User-agent: MetaInspector
4649  User-agent: MetaJobBot
50+ User-agent: MistralBot
4751  User-agent: netEstate Imprint Crawler
4852  User-agent: newspaper
4953  User-agent: Nutch
@@ -58,6 +62,7 @@
5862  User-agent: ServerHunterSpider
5963  User-agent: Spider
6064  User-agent: StatsDroneBot
65+ User-agent: Thinkbot
6166  User-agent: Timpibot
6267  User-agent: VelenPublicWebCrawler
6368  User-agent: webzio-extended
@@ -72,8 +77,10 @@
7277  User-agent: adidxbot
7378  User-agent: AdkernelTopicCrawler
7479  User-agent: admantx
80+ User-agent: AdmixerBot
7581  User-agent: AdsBot-Google
7682  User-agent: AdsBot-Google-Mobile
83+ User-agent: AffsignalCrawler
7784  User-agent: Ahrefs Site Audit
7885  User-agent: AhrefsBot
7986  User-agent: AhrefsSiteAudit
@@ -138,6 +145,7 @@
138145  User-agent: DomainStatsBot
139146  User-agent: DotBot
140147  User-agent: Dragonbot
148+ User-agent: Dragonfly
141149  User-agent: ds9
142150  User-agent: DVbot
143151  User-agent: Echobot Bot
@@ -154,6 +162,7 @@
154162  User-agent: Google AdSense
155163  User-agent: Google Digital Asset Links
156164  User-agent: Google StoreBot
165+ User-agent: Google-adstxt
157166  User-agent: Google-Adwords-Instant
158167  User-agent: Google-Display-Ads-Bot
159168  User-agent: Google-Structured-Data-Testing
@@ -165,6 +174,7 @@
165174  User-agent: hypestat
166175  User-agent: IAS crawler
167176  User-agent: ias-ie
177+ User-agent: IDG/IT
168178  User-agent: ImagesiftBot
169179  User-agent: Innguma
170180  User-agent: InteractiveAdvertisingBureau
@@ -176,6 +186,7 @@
176186  User-agent: Klaviyo
177187  User-agent: KStandBot
178188  User-agent: Leikibot
189+ User-agent: LibreCrawl
179190  User-agent: LightspeedSystemsCrawler
180191  User-agent: LinkCheck by Siteimprove
181192  User-agent: LinkChecker Bot
@@ -187,9 +198,11 @@
187198  User-agent: LinkWalker
188199  User-agent: Macrobondbot
189200  User-agent: magpie-crawler
201+ User-agent: Mantisbot
190202  User-agent: MarketGoo
191203  User-agent: MBCrawler
192204  User-agent: McontextualBot
205+ User-agent: mediaboardbot
193206  User-agent: MedialogiaBot
194207  User-agent: MediaMonitoringBot
195208  User-agent: Mediapartners-Google
@@ -213,6 +226,7 @@
213226  User-agent: Neticle Crawler
214227  User-agent: NetSeer crawler
215228  User-agent: Netvibes
229+ User-agent: NeuwoBot
216230  User-agent: NewsRoom.BI
217231  User-agent: Nitro-
218232  User-agent: NitroBot
@@ -238,6 +252,9 @@
238252  User-agent: Quantcastbot
239253  User-agent: QuuuBot
240254  User-agent: Qwarrybot
255+ User-agent: RankMath Link Checker
256+ User-agent: RankurBot
257+ User-agent: RavenCrawler
241258  User-agent: Readable
242259  User-agent: RetroListeCOM
243260  User-agent: RevvimGort
@@ -267,6 +284,7 @@
267284  User-agent: SEOkicks
268285  User-agent: SEOlizer
269286  User-agent: SERankingBacklinksBot
287+ User-agent: SerpMinerBot
270288  User-agent: serpstatbot
271289  User-agent: SiteAuditBot
272290  User-agent: Sitebulb
@@ -279,12 +297,14 @@
279297  User-agent: slickstream
280298  User-agent: SmartologyBot
281299  User-agent: SMTBot
300+ User-agent: Sottopop
282301  User-agent: StatistikAustria
283302  User-agent: StatusNestBacklinkSpider
284303  User-agent: Storebot-Google
285304  User-agent: Stripebot
286305  User-agent: Taboola
287306  User-agent: Taboolabot
307+ User-agent: TechSpyBot
288308  User-agent: trendictionbot
289309  User-agent: TrendsmapResolver
290310  User-agent: TTD-Content