NeuralCrawl

Slate / robots.txt snapshot

← back to slate.com · fetched 2026-08-25T19:41:43Z (24d ago) · HTTP 200 · 9052 bytes · sha256 51a818b38f5e2773 · raw

final URL: https://slate.com/robots.txt

1User-agent: *
2Disallow: /search
3Disallow: /comments/
4Disallow: /_
5
6Sitemap: https://slate.com/sitemaps/sitemap-index.xml
7
8User-agent: AmazonAdBot
9User-agent: Amazon-Advertising-ad-standards-bot
10Allow: /
11
12# Known agents
13
14# AI Data Scraper, Scraper
15User-agent: 008
16User-agent: AdsTxtCrawler
17User-agent: AI2Bot
18User-agent: Ai2Bot-Dolma
19User-agent: Amazonbot
20User-agent: Applebot-Extended
21User-agent: bedrockbot
22User-agent: Bytespider
23User-agent: CCBot
24User-agent: ChatGLM-Spider
25User-agent: ClaudeBot
26User-agent: CloudVertexBot
27User-agent: cohere-training-data-crawler
28User-agent: CohereBot
29User-agent: Cotoyogi
30User-agent: Dataprovider.com
31User-agent: Datenbank Crawler
32User-agent: dcrawl
33User-agent: DeepSeekBot
34User-agent: Doubaobot
35User-agent: ERNIEBot
36User-agent: FacebookBot
37User-agent: Google-Extended
38User-agent: GoogleOther
39User-agent: GPTBot
40User-agent: HelloworkJobPostingBot
41User-agent: HTTrack
42User-agent: HTTrack 3.0
43User-agent: ICC-Crawler
44User-agent: imageSpider
45User-agent: IndeedJobBot
46User-agent: Kangaroo Bot
47User-agent: KimiBot
48User-agent: KimiCrawler
49User-agent: KrawlerBot
50User-agent: laion-huggingface-processor
51User-agent: LCC
52User-agent: meta-externalagent
53User-agent: MetaInspector
54User-agent: MetaJobBot
55User-agent: micro-crawl
56User-agent: MistralBot
57User-agent: MoonshotBot
58User-agent: MoonSpider
59User-agent: netEstate Imprint Crawler
60User-agent: newspaper
61User-agent: Nutch
62User-agent: nyt_scraping
63User-agent: Offline Explorer
64User-agent: omgili
65User-agent: OpenindexSpider
66User-agent: PanguBot
67User-agent: Potions
68User-agent: QwenBot
69User-agent: Reflectionbot
70User-agent: SalamandraVLM
71User-agent: SBIntuitionsBot
72User-agent: Scrapy
73User-agent: ServerHunterSpider
74User-agent: Spider
75User-agent: StatsDroneBot
76User-agent: Thinkbot
77User-agent: Timpibot
78User-agent: VelenPublicWebCrawler
79User-agent: webzio-extended
80Disallow: /
81
82# Intelligence Gatherer, SEO Crawler
83User-agent: A360-Search
84User-agent: aa
85User-agent: ActiveComply
86User-agent: Adagiobot
87User-agent: adbeat_bot
88User-agent: adidxbot
89User-agent: AdkernelTopicCrawler
90User-agent: admantx
91User-agent: AdmixerBot
92User-agent: AdsBot-Google
93User-agent: AdsBot-Google-Mobile
94User-agent: AffsignalCrawler
95User-agent: Ahrefs Site Audit
96User-agent: AhrefsBot
97User-agent: AhrefsSiteAudit
98User-agent: aiHitBot
99User-agent: Amazing-SearchBot
100User-agent: AmazingTest-SearchBot
101User-agent: Amazon AdBot
102User-agent: AmazonProductDiscovery
103User-agent: AndersPinkBot
104User-agent: ArchiveBot
105User-agent: Assaybot
106User-agent: Attracta
107User-agent: AudigentAdBot
108User-agent: Audisto Crawler
109User-agent: Automattic Analytics Crawler
110User-agent: Awario
111User-agent: AwarioBot
112User-agent: AwarioSmartBot
113User-agent: Barkrowler
114User-agent: BigUpDataBot
115User-agent: Bing Ads
116User-agent: BitSightBot
117User-agent: Blackboard
118User-agent: BLEXBot
119User-agent: BomboraBot
120User-agent: botify
121User-agent: BrandVerity
122User-agent: BrightEdge Bot
123User-agent: BrightEdge Crawler
124User-agent: Buck
125User-agent: CaliberBot
126User-agent: Caliperbot
127User-agent: captify-crawler
128User-agent: channable
129User-agent: CheckMarkNetwork
130User-agent: Cincraw
131User-agent: ClarityBot
132User-agent: Clearscopebot
133User-agent: Clickagy Intelligence Bot
134User-agent: Clickagy Intelligence Bot v2
135User-agent: Cloudflare Custom Hostname Verification
136User-agent: Cloudflare Stream Webhook
137User-agent: Cloudflare-Healthchecks
138User-agent: CloudFlare-Prefetch
139User-agent: Cloudflare-SSLDetector
140User-agent: Cloudflare-Traffic-Manager
141User-agent: Cloudtrellis
142User-agent: cludo.com
143User-agent: Cocolyzebot
144User-agent: cognitiveSEO Crawler
145User-agent: contentking
146User-agent: ContextualBot
147User-agent: contxbot
148User-agent: Convermax
149User-agent: CriteoBot
150User-agent: Cxense
151User-agent: cXensebot
152User-agent: DataForSEO
153User-agent: DataForSEO Bot
154User-agent: DataForSeoBot
155User-agent: deepcrawl
156User-agent: deepcrawl.com
157User-agent: DomainStatsBot
158User-agent: DotBot
159User-agent: Dragonbot
160User-agent: Dragonfly
161User-agent: ds9
162User-agent: DVbot
163User-agent: Echobot Bot
164User-agent: EcoVadisSustainabilityBot
165User-agent: EpivozCrawler
166User-agent: ev-crawler
167User-agent: EzLynx
168User-agent: EzoicBot
169User-agent: EzoicBot-Nicheiq
170User-agent: factset_spyderbot
171User-agent: FDL Stats Bot
172User-agent: Fedicabot
173User-agent: GnowitNewsbot
174User-agent: Google AdsBot
175User-agent: Google AdSense
176User-agent: Google Digital Asset Links
177User-agent: Google StoreBot
178User-agent: Google-adstxt
179User-agent: Google-Adwords-Instant
180User-agent: Google-Display-Ads-Bot
181User-agent: Google-Structured-Data-Testing
182User-agent: GrapeshotCrawler
183User-agent: GroovinaAdsbot
184User-agent: HubSpot
185User-agent: Huckabot
186User-agent: Huckabuy Bot
187User-agent: hypestat
188User-agent: IAS crawler
189User-agent: ias-ie
190User-agent: IDG/IT
191User-agent: ImagesiftBot
192User-agent: Innguma
193User-agent: InteractiveAdvertisingBureau
194User-agent: IonCrawl
195User-agent: James BOT
196User-agent: Jugendschutzprogramm-Crawler
197User-agent: KargoBot
198User-agent: KargoBot-Artemis-Mobile
199User-agent: Klaviyo
200User-agent: KStandBot
201User-agent: Leikibot
202User-agent: LibreCrawl
203User-agent: LightspeedSystemsCrawler
204User-agent: LinkCheck by Siteimprove
205User-agent: LinkChecker Bot
206User-agent: linkchecker.pro
207User-agent: LinkCheckerBot
208User-agent: linkdexbot
209User-agent: linkfluence
210User-agent: LinksIndexerBot
211User-agent: LinkWalker
212User-agent: Macrobondbot
213User-agent: magpie-crawler
214User-agent: Mantisbot
215User-agent: MarketGoo
216User-agent: MBCrawler
217User-agent: McontextualBot
218User-agent: mediaboardbot
219User-agent: MedialogiaBot
220User-agent: MediaMonitoringBot
221User-agent: Mediapartners-Google
222User-agent: Mediatoolkitbot
223User-agent: Mediavine Medatada Parser
224User-agent: MediavineMetadataParser
225User-agent: MegaIndex.ru
226User-agent: meta-externalads
227User-agent: MgidBot
228User-agent: Missinglettr Bot
229User-agent: MixrankBot
230User-agent: MJ12bot
231User-agent: ModernizeBot
232User-agent: Moz dotbot
233User-agent: Moz rogerbot
234User-agent: MuckRack
235User-agent: NanoInteractive
236User-agent: Netcraft
237User-agent: NetcraftSurveyAgent
238User-agent: netEstate NE Crawler
239User-agent: Neticle Crawler
240User-agent: NetSeer crawler
241User-agent: Netvibes
242User-agent: NeuwoBot
243User-agent: NewsRoom.BI
244User-agent: Nitro-
245User-agent: NitroBot
246User-agent: noorobot
247User-agent: NostoCrawlerBot
248User-agent: OAI-AdsBot
249User-agent: OnCrawl
250User-agent: OnCrawl Mobile
251User-agent: online-webceo-bot
252User-agent: outboundbuddy-enrichment
253User-agent: outbrain
254User-agent: Owler
255User-agent: page-preview-tool
256User-agent: Pandalytics
257User-agent: panscient.com
258User-agent: parse.ly scraper
259User-agent: PartnerOptimizer-bot
260User-agent: peer39_crawler
261User-agent: Prerender
262User-agent: Pro Sitemaps
263User-agent: Pro Sitemaps Generator
264User-agent: productsup.io/crawler
265User-agent: proximic
266User-agent: Quantcastbot
267User-agent: QuuuBot
268User-agent: Qwarrybot
269User-agent: RankMath Link Checker
270User-agent: RankurBot
271User-agent: RavenCrawler
272User-agent: Readable
273User-agent: RetroListeCOM
274User-agent: RevvimGort
275User-agent: rogerbot-crawler
276User-agent: RSiteAuditor
277User-agent: RyteBot
278User-agent: SalesViewerBot
279User-agent: scoop.it
280User-agent: Scope3
281User-agent: [email protected]
282User-agent: Screaming Frog SEO Spider
283User-agent: SearchAtlas Bot
284User-agent: SEBot-WA
285User-agent: SemrushBot
286User-agent: SemrushBot-BA
287User-agent: SemrushBot-CT
288User-agent: SemrushBot-OCOB
289User-agent: SemrushBot-SA
290User-agent: SemrushBot-SI
291User-agent: SemrushBotBacklinks
292User-agent: SentiBot
293User-agent: Senutobot
294User-agent: seo-audit-check-bot
295User-agent: seo4ajax
296User-agent: seo4ajax.com
297User-agent: SeobilityBot
298User-agent: SEOkicks
299User-agent: SEOlizer
300User-agent: SERankingBacklinksBot
301User-agent: SerpMinerBot
302User-agent: serpstatbot
303User-agent: SirdataBot
304User-agent: SiteAuditBot
305User-agent: Sitebulb
306User-agent: SiteCheck-sitecrawl
307User-agent: SiteCheckerBotCrawler
308User-agent: SiteGuruCrawler
309User-agent: Siteimprove Crawl
310User-agent: Siteliner
311User-agent: SlickBot
312User-agent: slickstream
313User-agent: SmartologyBot
314User-agent: SMTBot
315User-agent: Sottopop
316User-agent: StatistikAustria
317User-agent: StatusNestBacklinkSpider
318User-agent: Storebot-Google
319User-agent: Stripebot
320User-agent: Taboola
321User-agent: Taboolabot
322User-agent: TechSpyBot
323User-agent: trendictionbot
324User-agent: TrendsmapResolver
325User-agent: TTD-Content
326User-agent: TurnitinBot
327User-agent: TweetmemeBot
328User-agent: Twingly
329User-agent: Twingly Recon-Sjostrom
330User-agent: uipbot
331User-agent: um-FC
332User-agent: um-IC
333User-agent: um-LN
334User-agent: UpContent
335User-agent: UpContent-QA
336User-agent: UpContent-Test
337User-agent: Verity
338User-agent: virustotal
339User-agent: W3 Validator Services
340User-agent: weborama-fetcher
341User-agent: webspidermount
342User-agent: WebTrackrCrawler
343User-agent: webzio
344User-agent: wknd-bot
345User-agent: woorankreview
346User-agent: XoviOnpageCrawler
347User-agent: YaDirectFetcher
348User-agent: Yahoo Ad monitoring
349User-agent: Yahoo Japan
350User-agent: YaK
351User-agent: Yext Inc
352User-agent: YextBot
353User-agent: Zoombot
354User-agent: ZoominfoBot
355Disallow: /search
356Disallow: /comments/
357Disallow: /_
358Disallow: /css
359Disallow: /fonts
360Disallow: /media
361Disallow: /piano
362Disallow: /static
363Crawl-delay: 10