NeuralCrawl

Slate / robots.txt snapshot

← back to slate.com · fetched 2026-10-13T23:51:59Z (24d ago) · HTTP 200 · 11372 bytes · sha256 4bad055a92718999 · raw

final URL: https://slate.com/robots.txt

1User-agent: *
2Disallow: /search
3Disallow: /comments/
4Disallow: /_
5
6Sitemap: https://slate.com/sitemaps/sitemap-index.xml
7
8User-agent: AmazonAdBot
9User-agent: Amazon-Advertising-ad-standards-bot
10Allow: /
11
12# Known agents
13
14# AI Data Scraper, Scraper
15User-agent: 008
16User-agent: AdsTxtCrawler
17User-agent: AgentDataBot
18User-agent: AI2Bot
19User-agent: Ai2Bot-Dolma
20User-agent: Amazonbot
21User-agent: ApifyBot
22User-agent: ApifyWebsiteContentCrawler
23User-agent: Applebot-Extended
24User-agent: BeansLLM-CorpusBot
25User-agent: bedrockbot
26User-agent: BixelBot
27User-agent: Bravebot
28User-agent: Brightbot
29User-agent: Bytespider
30User-agent: CCBot
31User-agent: ChatGLM-Spider
32User-agent: ClaudeBot
33User-agent: CloudflareBrowserRenderingCrawler
34User-agent: CloudVertexBot
35User-agent: cohere-training-data-crawler
36User-agent: CohereBot
37User-agent: Cotoyogi
38User-agent: CragCrawler
39User-agent: Crawlspace
40User-agent: Datenbank Crawler
41User-agent: dcrawl
42User-agent: DeepSeekBot
43User-agent: DesearchBot
44User-agent: Diffbot
45User-agent: Doubaobot
46User-agent: ERNIEBot
47User-agent: ExaBot
48User-agent: FacebookBot
49User-agent: FirecrawlAgent
50User-agent: Google-Extended
51User-agent: GoogleOther
52User-agent: GPTBot
53User-agent: HelloworkJobPostingBot
54User-agent: HenkBot
55User-agent: HTTrack
56User-agent: HTTrack 3.0
57User-agent: Hunyuan
58User-agent: ICC-Crawler
59User-agent: ImageBot
60User-agent: imageSpider
61User-agent: IndeedJobBot
62User-agent: Kangaroo Bot
63User-agent: Keenable-User
64User-agent: KeenableBot
65User-agent: KimiBot
66User-agent: KimiCrawler
67User-agent: KrawlerBot
68User-agent: laion-huggingface-processor
69User-agent: LCC
70User-agent: meta-externalagent
71User-agent: MetaInspector
72User-agent: MetaJobBot
73User-agent: micro-crawl
74User-agent: MistralBot
75User-agent: MoonshotBot
76User-agent: MoonSpider
77User-agent: Mozilla-Tabstack
78User-agent: netEstate Imprint Crawler
79User-agent: newspaper
80User-agent: Nutch
81User-agent: nyt_scraping
82User-agent: Offline Explorer
83User-agent: omgili
84User-agent: OpenindexSpider
85User-agent: PanguBot
86User-agent: Potions
87User-agent: Querit-SearchBot
88User-agent: QueritBot
89User-agent: QwenBot
90User-agent: recraft-image-dataset
91User-agent: ReflectionBot
92User-agent: SalamandraVLM
93User-agent: SBIntuitionsBot
94User-agent: Scrapy
95User-agent: ScryBot
96User-agent: ServerHunterSpider
97User-agent: ShapBot
98User-agent: SofyaBot
99User-agent: Spider
100User-agent: StatsDroneBot
101User-agent: TalarionFrontierCrawler
102User-agent: TavilyBot
103User-agent: Terra Cotta
104User-agent: TerraCotta
105User-agent: Thinkbot
106User-agent: Timpibot
107User-agent: VelenPublicWebCrawler
108User-agent: web-image-collection-research
109User-agent: webzio-extended
110User-agent: YiBot
111User-agent: YouBot
112Disallow: /
113
114# Intelligence Gatherer, SEO Crawler
115User-agent: A360-Search
116User-agent: aa
117User-agent: ActiveComply
118User-agent: Adagiobot
119User-agent: adbeat_bot
120User-agent: adidxbot
121User-agent: AdkernelTopicCrawler
122User-agent: admantx
123User-agent: AdmixerBot
124User-agent: AdsBot-Google
125User-agent: AdsBot-Google-Mobile
126User-agent: AffsignalCrawler
127User-agent: Ahrefs Site Audit
128User-agent: AhrefsBot
129User-agent: AhrefsSiteAudit
130User-agent: aiHitBot
131User-agent: Amazing-SearchBot
132User-agent: AmazingTest-SearchBot
133User-agent: Amazon AdBot
134User-agent: AmazonProductDiscovery
135User-agent: AndersPinkBot
136User-agent: ArchiveBot
137User-agent: Assaybot
138User-agent: Attracta
139User-agent: AudigentAdBot
140User-agent: Audisto Crawler
141User-agent: Automattic Analytics Crawler
142User-agent: AutotouchCompanyCrawler
143User-agent: Awario
144User-agent: AwarioBot
145User-agent: AwarioSmartBot
146User-agent: Barkrowler
147User-agent: BigUpDataBot
148User-agent: Bing Ads
149User-agent: BitSightBot
150User-agent: Blackboard
151User-agent: BLEXBot
152User-agent: BomboraBot
153User-agent: botify
154User-agent: BrandVerity
155User-agent: BrightEdge Bot
156User-agent: BrightEdge Crawler
157User-agent: Buck
158User-agent: CaliberBot
159User-agent: Caliperbot
160User-agent: CandidOrgWebCheck
161User-agent: captify-crawler
162User-agent: centryx-domain-mill
163User-agent: channable
164User-agent: CheckMarkNetwork
165User-agent: Cincraw
166User-agent: ClarityBot
167User-agent: Clearscopebot
168User-agent: Clickagy Intelligence Bot
169User-agent: Clickagy Intelligence Bot v2
170User-agent: Cloudflare Custom Hostname Verification
171User-agent: Cloudflare Stream Webhook
172User-agent: Cloudflare-Healthchecks
173User-agent: CloudFlare-Prefetch
174User-agent: Cloudflare-SSLDetector
175User-agent: Cloudflare-Traffic-Manager
176User-agent: Cloudtrellis
177User-agent: cludo.com
178User-agent: Cocolyzebot
179User-agent: cognitiveSEO Crawler
180User-agent: CompanyMapBot
181User-agent: contentking
182User-agent: ContextualBot
183User-agent: ContextWebCom-AdsTxt
184User-agent: contxbot
185User-agent: Convermax
186User-agent: CriteoBot
187User-agent: Cxense
188User-agent: cXensebot
189User-agent: CypressGrowthDiligence
190User-agent: DataForSEO
191User-agent: DataForSEO Bot
192User-agent: DataForSeoBot
193User-agent: Dataprovider
194User-agent: Dataprovider.com
195User-agent: deepcrawl
196User-agent: deepcrawl.com
197User-agent: DomainDrift
198User-agent: DomainStatsBot
199User-agent: DotBot
200User-agent: Dragonbot
201User-agent: Dragonfly
202User-agent: ds9
203User-agent: DVbot
204User-agent: dwelltime-bot
205User-agent: Echobot Bot
206User-agent: EcoVadisSustainabilityBot
207User-agent: EpivozCrawler
208User-agent: ev-crawler
209User-agent: EzLynx
210User-agent: EzoicBot
211User-agent: EzoicBot-Nicheiq
212User-agent: factset_spyderbot
213User-agent: FDL Stats Bot
214User-agent: Fedicabot
215User-agent: FirmCredBot
216User-agent: FlowIQLabsBot
217User-agent: ForestEngine
218User-agent: getdomaindata
219User-agent: GlobRadarBot
220User-agent: GnowitNewsbot
221User-agent: Google AdsBot
222User-agent: Google AdSense
223User-agent: Google Digital Asset Links
224User-agent: Google StoreBot
225User-agent: Google-adstxt
226User-agent: Google-Adwords-Instant
227User-agent: Google-Display-Ads-Bot
228User-agent: Google-Structured-Data-Testing
229User-agent: GovAPIEnricher
230User-agent: GrapeshotCrawler
231User-agent: GroovinaAdsbot
232User-agent: GrowthXBot
233User-agent: HubSpot
234User-agent: Huckabot
235User-agent: Huckabuy Bot
236User-agent: hypestat
237User-agent: IAS crawler
238User-agent: ias-ie
239User-agent: IDG/IT
240User-agent: ImagesiftBot
241User-agent: IncumbINTBot
242User-agent: Innguma
243User-agent: IntentBeamBot
244User-agent: InteractiveAdvertisingBureau
245User-agent: InterviewOS
246User-agent: IonCrawl
247User-agent: James BOT
248User-agent: Jugendschutzprogramm-Crawler
249User-agent: KargoBot
250User-agent: KargoBot-Artemis-Mobile
251User-agent: KimeBot
252User-agent: Klaviyo
253User-agent: Konturbot
254User-agent: KStandBot
255User-agent: LeadFinderBot
256User-agent: LeadForge
257User-agent: Leikibot
258User-agent: LibreCrawl
259User-agent: LightspeedSystemsCrawler
260User-agent: LinkCheck by Siteimprove
261User-agent: LinkChecker Bot
262User-agent: linkchecker.pro
263User-agent: LinkCheckerBot
264User-agent: linkdexbot
265User-agent: linkfluence
266User-agent: LinksIndexerBot
267User-agent: LinkWalker
268User-agent: ListSignalBot
269User-agent: Macrobondbot
270User-agent: magpie-crawler
271User-agent: Mantisbot
272User-agent: MapTheNetBot
273User-agent: MarketGoo
274User-agent: MBCrawler
275User-agent: McontextualBot
276User-agent: mediaboardbot
277User-agent: MedialogiaBot
278User-agent: MediaMonitoringBot
279User-agent: Mediapartners-Google
280User-agent: Mediatoolkitbot
281User-agent: Mediavine Medatada Parser
282User-agent: MediavineMetadataParser
283User-agent: MegaIndex.ru
284User-agent: meta-externalads
285User-agent: MgidBot
286User-agent: Missinglettr Bot
287User-agent: MixrankBot
288User-agent: MJ12bot
289User-agent: ModernizeBot
290User-agent: Moz dotbot
291User-agent: Moz rogerbot
292User-agent: MRLeadsBot
293User-agent: MuckRack
294User-agent: NanoInteractive
295User-agent: Netcraft
296User-agent: NetcraftSurveyAgent
297User-agent: netEstate NE Crawler
298User-agent: Neticle Crawler
299User-agent: NetSeer crawler
300User-agent: Netvibes
301User-agent: NeuwoBot
302User-agent: NewsRoom.BI
303User-agent: Nitro-
304User-agent: NitroBot
305User-agent: NoahBot
306User-agent: noorobot
307User-agent: NostoCrawlerBot
308User-agent: OAI-AdsBot
309User-agent: OmvionLeadLake
310User-agent: OnCrawl
311User-agent: OnCrawl Mobile
312User-agent: online-webceo-bot
313User-agent: osentix-crawler
314User-agent: outboundbuddy-enrichment
315User-agent: outbrain
316User-agent: Owler
317User-agent: page-preview-tool
318User-agent: Pandalytics
319User-agent: panscient.com
320User-agent: parse.ly scraper
321User-agent: PartnerOptimizer-bot
322User-agent: peer39_crawler
323User-agent: PipericBot
324User-agent: PostGooBot
325User-agent: PostitLeadDiscovery
326User-agent: PoweredByBot
327User-agent: Pro Sitemaps
328User-agent: Pro Sitemaps Generator
329User-agent: productsup.io/crawler
330User-agent: PrognostiXonThinMiner
331User-agent: proximic
332User-agent: QlyzeBot
333User-agent: Quantcastbot
334User-agent: QuuuBot
335User-agent: Qwairy-Bot
336User-agent: Qwarrybot
337User-agent: RankMath Link Checker
338User-agent: RankshiftFetcher
339User-agent: RankurBot
340User-agent: RavenCrawler
341User-agent: Readable
342User-agent: RetroListeCOM
343User-agent: revops-nuvo-detector
344User-agent: RevvimGort
345User-agent: rogerbot-crawler
346User-agent: RSiteAuditor
347User-agent: RyteBot
348User-agent: SalesViewerBot
349User-agent: ScalinthBot
350User-agent: scoop.it
351User-agent: Scope3
352User-agent: [email protected]
353User-agent: Screaming Frog SEO Spider
354User-agent: SearchAtlas Bot
355User-agent: SEBot-WA
356User-agent: semantic-visions-analytica
357User-agent: semantic-visions-infoseeker
358User-agent: SemanticVisionsBot
359User-agent: SemrushBot
360User-agent: SemrushBot-BA
361User-agent: SemrushBot-CT
362User-agent: SemrushBot-OCOB
363User-agent: SemrushBot-SA
364User-agent: SemrushBot-SI
365User-agent: SemrushBotBacklinks
366User-agent: SenderBot
367User-agent: SentiBot
368User-agent: Senutobot
369User-agent: seo-audit-check-bot
370User-agent: seo4ajax
371User-agent: seo4ajax.com
372User-agent: SeobilityBot
373User-agent: SEOJuice-SearchBot
374User-agent: SEOkicks
375User-agent: SEOlizer
376User-agent: SEOToolsOutreach
377User-agent: SERankingBacklinksBot
378User-agent: SerpMinerBot
379User-agent: serpstatbot
380User-agent: SirdataBot
381User-agent: SiteAuditBot
382User-agent: Sitebulb
383User-agent: SiteCheck-sitecrawl
384User-agent: SiteCheckerBotCrawler
385User-agent: SiteGuruCrawler
386User-agent: Siteimprove Crawl
387User-agent: Siteliner
388User-agent: SiteStatsBot
389User-agent: SlickBot
390User-agent: slickstream
391User-agent: SmartologyBot
392User-agent: SMBsBot
393User-agent: SMTBot
394User-agent: Sottopop
395User-agent: SpamCipherBot
396User-agent: Speroll-AdsTxt-Crawler
397User-agent: StatistikAustria
398User-agent: StatusNestBacklinkSpider
399User-agent: Storebot-Google
400User-agent: StormIntelCrawler
401User-agent: Stripebot
402User-agent: Taboola
403User-agent: Taboolabot
404User-agent: TechSpyBot
405User-agent: TheWebReport
406User-agent: TitanBot
407User-agent: topicsearch-crawler
408User-agent: tphotobot
409User-agent: trendictionbot
410User-agent: TrendsmapResolver
411User-agent: TTD-Content
412User-agent: TurnitinBot
413User-agent: TweetmemeBot
414User-agent: Twingly
415User-agent: Twingly Recon-Sjostrom
416User-agent: uipbot
417User-agent: um-FC
418User-agent: um-IC
419User-agent: um-LN
420User-agent: UpContent
421User-agent: UpContent-QA
422User-agent: UpContent-Test
423User-agent: Verity
424User-agent: virustotal
425User-agent: W3 Validator Services
426User-agent: weborama-fetcher
427User-agent: webspidermount
428User-agent: WebTrackrCrawler
429User-agent: webzio
430User-agent: WhiteCircleTrustBot
431User-agent: WhyNoIPv6Bot
432User-agent: wknd-bot
433User-agent: woorankreview
434User-agent: XoviOnpageCrawler
435User-agent: YaDirectFetcher
436User-agent: Yahoo Ad monitoring
437User-agent: Yahoo Japan
438User-agent: YaK
439User-agent: Yext Inc
440User-agent: YextBot
441User-agent: Zoombot
442User-agent: ZoominfoBot
443Disallow: /search
444Disallow: /comments/
445Disallow: /_
446Disallow: /css
447Disallow: /fonts
448Disallow: /media
449Disallow: /piano
450Disallow: /static
451Crawl-delay: 10