NeuralCrawl

Slate / robots.txt

← back to slate.com · change detected 4d ago (2026-11-03T00:58:14Z) · modified +9 −0

What changed

Diff old (2026-10-13T23:51:59Z) → new (2026-11-03T00:58:14Z)

@@ -21,6 +21,7 @@
2121  User-agent: ApifyBot
2222  User-agent: ApifyWebsiteContentCrawler
2323  User-agent: Applebot-Extended
24+ User-agent: artsdata-crawler
2425  User-agent: BeansLLM-CorpusBot
2526  User-agent: bedrockbot
2627  User-agent: BixelBot
@@ -43,6 +44,7 @@
4344  User-agent: DesearchBot
4445  User-agent: Diffbot
4546  User-agent: Doubaobot
47+ User-agent: ElevenlabsCrawler
4648  User-agent: ERNIEBot
4749  User-agent: ExaBot
4850  User-agent: FacebookBot
@@ -84,6 +86,7 @@
8486  User-agent: OpenindexSpider
8587  User-agent: PanguBot
8688  User-agent: Potions
89+ User-agent: Quartr Crawler
8790  User-agent: Querit-SearchBot
8891  User-agent: QueritBot
8992  User-agent: QwenBot
@@ -121,6 +124,7 @@
121124  User-agent: AdkernelTopicCrawler
122125  User-agent: admantx
123126  User-agent: AdmixerBot
127+ User-agent: AdpResearchBot
124128  User-agent: AdsBot-Google
125129  User-agent: AdsBot-Google-Mobile
126130  User-agent: AffsignalCrawler
@@ -155,6 +159,7 @@
155159  User-agent: BrightEdge Bot
156160  User-agent: BrightEdge Crawler
157161  User-agent: Buck
162+ User-agent: BuiltWith
158163  User-agent: CaliberBot
159164  User-agent: Caliperbot
160165  User-agent: CandidOrgWebCheck
@@ -214,6 +219,7 @@
214219  User-agent: Fedicabot
215220  User-agent: FirmCredBot
216221  User-agent: FlowIQLabsBot
222+ User-agent: FlyingPress
217223  User-agent: ForestEngine
218224  User-agent: getdomaindata
219225  User-agent: GlobRadarBot
@@ -394,6 +400,7 @@
394400  User-agent: Sottopop
395401  User-agent: SpamCipherBot
396402  User-agent: Speroll-AdsTxt-Crawler
403+ User-agent: SpyglassesBot
397404  User-agent: StatistikAustria
398405  User-agent: StatusNestBacklinkSpider
399406  User-agent: Storebot-Google
@@ -423,6 +430,7 @@
423430  User-agent: Verity
424431  User-agent: virustotal
425432  User-agent: W3 Validator Services
433+ User-agent: WappalyzerBot
426434  User-agent: weborama-fetcher
427435  User-agent: webspidermount
428436  User-agent: WebTrackrCrawler
@@ -440,6 +448,7 @@
440448  User-agent: YextBot
441449  User-agent: Zoombot
442450  User-agent: ZoominfoBot
451+ User-agent: Zvelo
443452  Disallow: /search
444453  Disallow: /comments/
445454  Disallow: /_