NeuralCrawl

Slate / robots.txt

← back to slate.com · change detected 1mo ago (2026-09-29T18:55:02Z) · modified +70 −1

What changed

Diff old (2026-09-17T21:12:52Z) → new (2026-09-29T18:55:02Z)

@@ -14,19 +14,23 @@
1414  # AI Data Scraper, Scraper
1515  User-agent: 008
1616  User-agent: AdsTxtCrawler
17+ User-agent: AgentDataBot
1718  User-agent: AI2Bot
1819  User-agent: Ai2Bot-Dolma
1920  User-agent: Amazonbot
2021  User-agent: ApifyBot
2122  User-agent: ApifyWebsiteContentCrawler
2223  User-agent: Applebot-Extended
24+ User-agent: BeansLLM-CorpusBot
2325  User-agent: bedrockbot
26+ User-agent: BixelBot
2427  User-agent: Bravebot
2528  User-agent: Brightbot
2629  User-agent: Bytespider
2730  User-agent: CCBot
2831  User-agent: ChatGLM-Spider
2932  User-agent: ClaudeBot
33+ User-agent: CloudflareBrowserRenderingCrawler
3034  User-agent: CloudVertexBot
3135  User-agent: cohere-training-data-crawler
3236  User-agent: CohereBot
@@ -36,6 +40,7 @@
3640  User-agent: Datenbank Crawler
3741  User-agent: dcrawl
3842  User-agent: DeepSeekBot
43+ User-agent: DesearchBot
3944  User-agent: Diffbot
4045  User-agent: Doubaobot
4146  User-agent: ERNIEBot
@@ -49,10 +54,13 @@
4954  User-agent: HenkBot
5055  User-agent: HTTrack
5156  User-agent: HTTrack 3.0
57+ User-agent: Hunyuan
5258  User-agent: ICC-Crawler
5359  User-agent: imageSpider
5460  User-agent: IndeedJobBot
5561  User-agent: Kangaroo Bot
62+ User-agent: Keenable-User
63+ User-agent: KeenableBot
5664  User-agent: KimiBot
5765  User-agent: KimiCrawler
5866  User-agent: KrawlerBot
@@ -78,21 +86,27 @@
7886  User-agent: Querit-SearchBot
7987  User-agent: QueritBot
8088  User-agent: QwenBot
81- User-agent: Reflectionbot
89+ User-agent: recraft-image-dataset
90+ User-agent: ReflectionBot
8291  User-agent: SalamandraVLM
8392  User-agent: SBIntuitionsBot
8493  User-agent: Scrapy
94+ User-agent: ScryBot
8595  User-agent: ServerHunterSpider
8696  User-agent: ShapBot
97+ User-agent: SofyaBot
8798  User-agent: Spider
8899  User-agent: StatsDroneBot
100+ User-agent: TalarionFrontierCrawler
89101  User-agent: TavilyBot
90102  User-agent: Terra Cotta
91103  User-agent: TerraCotta
92104  User-agent: Thinkbot
93105  User-agent: Timpibot
94106  User-agent: VelenPublicWebCrawler
107+ User-agent: web-image-collection-research
95108  User-agent: webzio-extended
109+ User-agent: YiBot
96110  User-agent: YouBot
97111  Disallow: /
98112 
@@ -124,6 +138,7 @@
124138  User-agent: AudigentAdBot
125139  User-agent: Audisto Crawler
126140  User-agent: Automattic Analytics Crawler
141+ User-agent: AutotouchCompanyCrawler
127142  User-agent: Awario
128143  User-agent: AwarioBot
129144  User-agent: AwarioSmartBot
@@ -141,7 +156,9 @@
141156  User-agent: Buck
142157  User-agent: CaliberBot
143158  User-agent: Caliperbot
159+ User-agent: CandidOrgWebCheck
144160  User-agent: captify-crawler
161+ User-agent: centryx-domain-mill
145162  User-agent: channable
146163  User-agent: CheckMarkNetwork
147164  User-agent: Cincraw
@@ -159,13 +176,16 @@
159176  User-agent: cludo.com
160177  User-agent: Cocolyzebot
161178  User-agent: cognitiveSEO Crawler
179+ User-agent: CompanyMapBot
162180  User-agent: contentking
163181  User-agent: ContextualBot
182+ User-agent: ContextWebCom-AdsTxt
164183  User-agent: contxbot
165184  User-agent: Convermax
166185  User-agent: CriteoBot
167186  User-agent: Cxense
168187  User-agent: cXensebot
188+ User-agent: CypressGrowthDiligence
169189  User-agent: DataForSEO
170190  User-agent: DataForSEO Bot
171191  User-agent: DataForSeoBot
@@ -173,12 +193,14 @@
173193  User-agent: Dataprovider.com
174194  User-agent: deepcrawl
175195  User-agent: deepcrawl.com
196+ User-agent: DomainDrift
176197  User-agent: DomainStatsBot
177198  User-agent: DotBot
178199  User-agent: Dragonbot
179200  User-agent: Dragonfly
180201  User-agent: ds9
181202  User-agent: DVbot
203+ User-agent: dwelltime-bot
182204  User-agent: Echobot Bot
183205  User-agent: EcoVadisSustainabilityBot
184206  User-agent: EpivozCrawler
@@ -189,6 +211,11 @@
189211  User-agent: factset_spyderbot
190212  User-agent: FDL Stats Bot
191213  User-agent: Fedicabot
214+ User-agent: FirmCredBot
215+ User-agent: FlowIQLabsBot
216+ User-agent: ForestEngine
217+ User-agent: getdomaindata
218+ User-agent: GlobRadarBot
192219  User-agent: GnowitNewsbot
193220  User-agent: Google AdsBot
194221  User-agent: Google AdSense
@@ -198,8 +225,10 @@
198225  User-agent: Google-Adwords-Instant
199226  User-agent: Google-Display-Ads-Bot
200227  User-agent: Google-Structured-Data-Testing
228+ User-agent: GovAPIEnricher
201229  User-agent: GrapeshotCrawler
202230  User-agent: GroovinaAdsbot
231+ User-agent: GrowthXBot
203232  User-agent: HubSpot
204233  User-agent: Huckabot
205234  User-agent: Huckabuy Bot
@@ -208,15 +237,22 @@
208237  User-agent: ias-ie
209238  User-agent: IDG/IT
210239  User-agent: ImagesiftBot
240+ User-agent: IncumbINTBot
211241  User-agent: Innguma
242+ User-agent: IntentBeamBot
212243  User-agent: InteractiveAdvertisingBureau
244+ User-agent: InterviewOS
213245  User-agent: IonCrawl
214246  User-agent: James BOT
215247  User-agent: Jugendschutzprogramm-Crawler
216248  User-agent: KargoBot
217249  User-agent: KargoBot-Artemis-Mobile
250+ User-agent: KimeBot
218251  User-agent: Klaviyo
252+ User-agent: Konturbot
219253  User-agent: KStandBot
254+ User-agent: LeadFinderBot
255+ User-agent: LeadForge
220256  User-agent: Leikibot
221257  User-agent: LibreCrawl
222258  User-agent: LightspeedSystemsCrawler
@@ -228,9 +264,11 @@
228264  User-agent: linkfluence
229265  User-agent: LinksIndexerBot
230266  User-agent: LinkWalker
267+ User-agent: ListSignalBot
231268  User-agent: Macrobondbot
232269  User-agent: magpie-crawler
233270  User-agent: Mantisbot
271+ User-agent: MapTheNetBot
234272  User-agent: MarketGoo
235273  User-agent: MBCrawler
236274  User-agent: McontextualBot
@@ -250,6 +288,7 @@
250288  User-agent: ModernizeBot
251289  User-agent: Moz dotbot
252290  User-agent: Moz rogerbot
291+ User-agent: MRLeadsBot
253292  User-agent: MuckRack
254293  User-agent: NanoInteractive
255294  User-agent: Netcraft
@@ -262,12 +301,15 @@
262301  User-agent: NewsRoom.BI
263302  User-agent: Nitro-
264303  User-agent: NitroBot
304+ User-agent: NoahBot
265305  User-agent: noorobot
266306  User-agent: NostoCrawlerBot
267307  User-agent: OAI-AdsBot
308+ User-agent: OmvionLeadLake
268309  User-agent: OnCrawl
269310  User-agent: OnCrawl Mobile
270311  User-agent: online-webceo-bot
312+ User-agent: osentix-crawler
271313  User-agent: outboundbuddy-enrichment
272314  User-agent: outbrain
273315  User-agent: Owler
@@ -277,29 +319,42 @@
277319  User-agent: parse.ly scraper
278320  User-agent: PartnerOptimizer-bot
279321  User-agent: peer39_crawler
322+ User-agent: PipericBot
323+ User-agent: PostGooBot
324+ User-agent: PostitLeadDiscovery
325+ User-agent: PoweredByBot
280326  User-agent: Pro Sitemaps
281327  User-agent: Pro Sitemaps Generator
282328  User-agent: productsup.io/crawler
329+ User-agent: PrognostiXonThinMiner
283330  User-agent: proximic
331+ User-agent: QlyzeBot
284332  User-agent: Quantcastbot
285333  User-agent: QuuuBot
334+ User-agent: Qwairy-Bot
286335  User-agent: Qwarrybot
287336  User-agent: RankMath Link Checker
337+ User-agent: RankshiftFetcher
288338  User-agent: RankurBot
289339  User-agent: RavenCrawler
290340  User-agent: Readable
291341  User-agent: RetroListeCOM
342+ User-agent: revops-nuvo-detector
292343  User-agent: RevvimGort
293344  User-agent: rogerbot-crawler
294345  User-agent: RSiteAuditor
295346  User-agent: RyteBot
296347  User-agent: SalesViewerBot
348+ User-agent: ScalinthBot
297349  User-agent: scoop.it
298350  User-agent: Scope3
299351  User-agent: [email protected]
300352  User-agent: Screaming Frog SEO Spider
301353  User-agent: SearchAtlas Bot
302354  User-agent: SEBot-WA
355+ User-agent: semantic-visions-analytica
356+ User-agent: semantic-visions-infoseeker
357+ User-agent: SemanticVisionsBot
303358  User-agent: SemrushBot
304359  User-agent: SemrushBot-BA
305360  User-agent: SemrushBot-CT
@@ -307,14 +362,17 @@
307362  User-agent: SemrushBot-SA
308363  User-agent: SemrushBot-SI
309364  User-agent: SemrushBotBacklinks
365+ User-agent: SenderBot
310366  User-agent: SentiBot
311367  User-agent: Senutobot
312368  User-agent: seo-audit-check-bot
313369  User-agent: seo4ajax
314370  User-agent: seo4ajax.com
315371  User-agent: SeobilityBot
372+ User-agent: SEOJuice-SearchBot
316373  User-agent: SEOkicks
317374  User-agent: SEOlizer
375+ User-agent: SEOToolsOutreach
318376  User-agent: SERankingBacklinksBot
319377  User-agent: SerpMinerBot
320378  User-agent: serpstatbot
@@ -326,18 +384,27 @@
326384  User-agent: SiteGuruCrawler
327385  User-agent: Siteimprove Crawl
328386  User-agent: Siteliner
387+ User-agent: SiteStatsBot
329388  User-agent: SlickBot
330389  User-agent: slickstream
331390  User-agent: SmartologyBot
391+ User-agent: SMBsBot
332392  User-agent: SMTBot
333393  User-agent: Sottopop
394+ User-agent: SpamCipherBot
395+ User-agent: Speroll-AdsTxt-Crawler
334396  User-agent: StatistikAustria
335397  User-agent: StatusNestBacklinkSpider
336398  User-agent: Storebot-Google
399+ User-agent: StormIntelCrawler
337400  User-agent: Stripebot
338401  User-agent: Taboola
339402  User-agent: Taboolabot
340403  User-agent: TechSpyBot
404+ User-agent: TheWebReport
405+ User-agent: TitanBot
406+ User-agent: topicsearch-crawler
407+ User-agent: tphotobot
341408  User-agent: trendictionbot
342409  User-agent: TrendsmapResolver
343410  User-agent: TTD-Content
@@ -359,6 +426,8 @@
359426  User-agent: webspidermount
360427  User-agent: WebTrackrCrawler
361428  User-agent: webzio
429+ User-agent: WhiteCircleTrustBot
430+ User-agent: WhyNoIPv6Bot
362431  User-agent: wknd-bot
363432  User-agent: woorankreview
364433  User-agent: XoviOnpageCrawler