NeuralCrawl

Slate / robots.txt snapshot

← back to slate.com · fetched 2026-09-17T21:12:52Z (1d ago) · HTTP 200 · 9479 bytes · sha256 454f4f6491b00b4e · raw

final URL: https://slate.com/robots.txt

1User-agent: *
2Disallow: /search
3Disallow: /comments/
4Disallow: /_
5
6Sitemap: https://slate.com/sitemaps/sitemap-index.xml
7
8User-agent: AmazonAdBot
9User-agent: Amazon-Advertising-ad-standards-bot
10Allow: /
11
12# Known agents
13
14# AI Data Scraper, Scraper
15User-agent: 008
16User-agent: AdsTxtCrawler
17User-agent: AI2Bot
18User-agent: Ai2Bot-Dolma
19User-agent: Amazonbot
20User-agent: ApifyBot
21User-agent: ApifyWebsiteContentCrawler
22User-agent: Applebot-Extended
23User-agent: bedrockbot
24User-agent: Bravebot
25User-agent: Brightbot
26User-agent: Bytespider
27User-agent: CCBot
28User-agent: ChatGLM-Spider
29User-agent: ClaudeBot
30User-agent: CloudVertexBot
31User-agent: cohere-training-data-crawler
32User-agent: CohereBot
33User-agent: Cotoyogi
34User-agent: CragCrawler
35User-agent: Crawlspace
36User-agent: Datenbank Crawler
37User-agent: dcrawl
38User-agent: DeepSeekBot
39User-agent: Diffbot
40User-agent: Doubaobot
41User-agent: ERNIEBot
42User-agent: ExaBot
43User-agent: FacebookBot
44User-agent: FirecrawlAgent
45User-agent: Google-Extended
46User-agent: GoogleOther
47User-agent: GPTBot
48User-agent: HelloworkJobPostingBot
49User-agent: HenkBot
50User-agent: HTTrack
51User-agent: HTTrack 3.0
52User-agent: ICC-Crawler
53User-agent: imageSpider
54User-agent: IndeedJobBot
55User-agent: Kangaroo Bot
56User-agent: KimiBot
57User-agent: KimiCrawler
58User-agent: KrawlerBot
59User-agent: laion-huggingface-processor
60User-agent: LCC
61User-agent: meta-externalagent
62User-agent: MetaInspector
63User-agent: MetaJobBot
64User-agent: micro-crawl
65User-agent: MistralBot
66User-agent: MoonshotBot
67User-agent: MoonSpider
68User-agent: Mozilla-Tabstack
69User-agent: netEstate Imprint Crawler
70User-agent: newspaper
71User-agent: Nutch
72User-agent: nyt_scraping
73User-agent: Offline Explorer
74User-agent: omgili
75User-agent: OpenindexSpider
76User-agent: PanguBot
77User-agent: Potions
78User-agent: Querit-SearchBot
79User-agent: QueritBot
80User-agent: QwenBot
81User-agent: Reflectionbot
82User-agent: SalamandraVLM
83User-agent: SBIntuitionsBot
84User-agent: Scrapy
85User-agent: ServerHunterSpider
86User-agent: ShapBot
87User-agent: Spider
88User-agent: StatsDroneBot
89User-agent: TavilyBot
90User-agent: Terra Cotta
91User-agent: TerraCotta
92User-agent: Thinkbot
93User-agent: Timpibot
94User-agent: VelenPublicWebCrawler
95User-agent: webzio-extended
96User-agent: YouBot
97Disallow: /
98
99# Intelligence Gatherer, SEO Crawler
100User-agent: A360-Search
101User-agent: aa
102User-agent: ActiveComply
103User-agent: Adagiobot
104User-agent: adbeat_bot
105User-agent: adidxbot
106User-agent: AdkernelTopicCrawler
107User-agent: admantx
108User-agent: AdmixerBot
109User-agent: AdsBot-Google
110User-agent: AdsBot-Google-Mobile
111User-agent: AffsignalCrawler
112User-agent: Ahrefs Site Audit
113User-agent: AhrefsBot
114User-agent: AhrefsSiteAudit
115User-agent: aiHitBot
116User-agent: Amazing-SearchBot
117User-agent: AmazingTest-SearchBot
118User-agent: Amazon AdBot
119User-agent: AmazonProductDiscovery
120User-agent: AndersPinkBot
121User-agent: ArchiveBot
122User-agent: Assaybot
123User-agent: Attracta
124User-agent: AudigentAdBot
125User-agent: Audisto Crawler
126User-agent: Automattic Analytics Crawler
127User-agent: Awario
128User-agent: AwarioBot
129User-agent: AwarioSmartBot
130User-agent: Barkrowler
131User-agent: BigUpDataBot
132User-agent: Bing Ads
133User-agent: BitSightBot
134User-agent: Blackboard
135User-agent: BLEXBot
136User-agent: BomboraBot
137User-agent: botify
138User-agent: BrandVerity
139User-agent: BrightEdge Bot
140User-agent: BrightEdge Crawler
141User-agent: Buck
142User-agent: CaliberBot
143User-agent: Caliperbot
144User-agent: captify-crawler
145User-agent: channable
146User-agent: CheckMarkNetwork
147User-agent: Cincraw
148User-agent: ClarityBot
149User-agent: Clearscopebot
150User-agent: Clickagy Intelligence Bot
151User-agent: Clickagy Intelligence Bot v2
152User-agent: Cloudflare Custom Hostname Verification
153User-agent: Cloudflare Stream Webhook
154User-agent: Cloudflare-Healthchecks
155User-agent: CloudFlare-Prefetch
156User-agent: Cloudflare-SSLDetector
157User-agent: Cloudflare-Traffic-Manager
158User-agent: Cloudtrellis
159User-agent: cludo.com
160User-agent: Cocolyzebot
161User-agent: cognitiveSEO Crawler
162User-agent: contentking
163User-agent: ContextualBot
164User-agent: contxbot
165User-agent: Convermax
166User-agent: CriteoBot
167User-agent: Cxense
168User-agent: cXensebot
169User-agent: DataForSEO
170User-agent: DataForSEO Bot
171User-agent: DataForSeoBot
172User-agent: Dataprovider
173User-agent: Dataprovider.com
174User-agent: deepcrawl
175User-agent: deepcrawl.com
176User-agent: DomainStatsBot
177User-agent: DotBot
178User-agent: Dragonbot
179User-agent: Dragonfly
180User-agent: ds9
181User-agent: DVbot
182User-agent: Echobot Bot
183User-agent: EcoVadisSustainabilityBot
184User-agent: EpivozCrawler
185User-agent: ev-crawler
186User-agent: EzLynx
187User-agent: EzoicBot
188User-agent: EzoicBot-Nicheiq
189User-agent: factset_spyderbot
190User-agent: FDL Stats Bot
191User-agent: Fedicabot
192User-agent: GnowitNewsbot
193User-agent: Google AdsBot
194User-agent: Google AdSense
195User-agent: Google Digital Asset Links
196User-agent: Google StoreBot
197User-agent: Google-adstxt
198User-agent: Google-Adwords-Instant
199User-agent: Google-Display-Ads-Bot
200User-agent: Google-Structured-Data-Testing
201User-agent: GrapeshotCrawler
202User-agent: GroovinaAdsbot
203User-agent: HubSpot
204User-agent: Huckabot
205User-agent: Huckabuy Bot
206User-agent: hypestat
207User-agent: IAS crawler
208User-agent: ias-ie
209User-agent: IDG/IT
210User-agent: ImagesiftBot
211User-agent: Innguma
212User-agent: InteractiveAdvertisingBureau
213User-agent: IonCrawl
214User-agent: James BOT
215User-agent: Jugendschutzprogramm-Crawler
216User-agent: KargoBot
217User-agent: KargoBot-Artemis-Mobile
218User-agent: Klaviyo
219User-agent: KStandBot
220User-agent: Leikibot
221User-agent: LibreCrawl
222User-agent: LightspeedSystemsCrawler
223User-agent: LinkCheck by Siteimprove
224User-agent: LinkChecker Bot
225User-agent: linkchecker.pro
226User-agent: LinkCheckerBot
227User-agent: linkdexbot
228User-agent: linkfluence
229User-agent: LinksIndexerBot
230User-agent: LinkWalker
231User-agent: Macrobondbot
232User-agent: magpie-crawler
233User-agent: Mantisbot
234User-agent: MarketGoo
235User-agent: MBCrawler
236User-agent: McontextualBot
237User-agent: mediaboardbot
238User-agent: MedialogiaBot
239User-agent: MediaMonitoringBot
240User-agent: Mediapartners-Google
241User-agent: Mediatoolkitbot
242User-agent: Mediavine Medatada Parser
243User-agent: MediavineMetadataParser
244User-agent: MegaIndex.ru
245User-agent: meta-externalads
246User-agent: MgidBot
247User-agent: Missinglettr Bot
248User-agent: MixrankBot
249User-agent: MJ12bot
250User-agent: ModernizeBot
251User-agent: Moz dotbot
252User-agent: Moz rogerbot
253User-agent: MuckRack
254User-agent: NanoInteractive
255User-agent: Netcraft
256User-agent: NetcraftSurveyAgent
257User-agent: netEstate NE Crawler
258User-agent: Neticle Crawler
259User-agent: NetSeer crawler
260User-agent: Netvibes
261User-agent: NeuwoBot
262User-agent: NewsRoom.BI
263User-agent: Nitro-
264User-agent: NitroBot
265User-agent: noorobot
266User-agent: NostoCrawlerBot
267User-agent: OAI-AdsBot
268User-agent: OnCrawl
269User-agent: OnCrawl Mobile
270User-agent: online-webceo-bot
271User-agent: outboundbuddy-enrichment
272User-agent: outbrain
273User-agent: Owler
274User-agent: page-preview-tool
275User-agent: Pandalytics
276User-agent: panscient.com
277User-agent: parse.ly scraper
278User-agent: PartnerOptimizer-bot
279User-agent: peer39_crawler
280User-agent: Pro Sitemaps
281User-agent: Pro Sitemaps Generator
282User-agent: productsup.io/crawler
283User-agent: proximic
284User-agent: Quantcastbot
285User-agent: QuuuBot
286User-agent: Qwarrybot
287User-agent: RankMath Link Checker
288User-agent: RankurBot
289User-agent: RavenCrawler
290User-agent: Readable
291User-agent: RetroListeCOM
292User-agent: RevvimGort
293User-agent: rogerbot-crawler
294User-agent: RSiteAuditor
295User-agent: RyteBot
296User-agent: SalesViewerBot
297User-agent: scoop.it
298User-agent: Scope3
299User-agent: [email protected]
300User-agent: Screaming Frog SEO Spider
301User-agent: SearchAtlas Bot
302User-agent: SEBot-WA
303User-agent: SemrushBot
304User-agent: SemrushBot-BA
305User-agent: SemrushBot-CT
306User-agent: SemrushBot-OCOB
307User-agent: SemrushBot-SA
308User-agent: SemrushBot-SI
309User-agent: SemrushBotBacklinks
310User-agent: SentiBot
311User-agent: Senutobot
312User-agent: seo-audit-check-bot
313User-agent: seo4ajax
314User-agent: seo4ajax.com
315User-agent: SeobilityBot
316User-agent: SEOkicks
317User-agent: SEOlizer
318User-agent: SERankingBacklinksBot
319User-agent: SerpMinerBot
320User-agent: serpstatbot
321User-agent: SirdataBot
322User-agent: SiteAuditBot
323User-agent: Sitebulb
324User-agent: SiteCheck-sitecrawl
325User-agent: SiteCheckerBotCrawler
326User-agent: SiteGuruCrawler
327User-agent: Siteimprove Crawl
328User-agent: Siteliner
329User-agent: SlickBot
330User-agent: slickstream
331User-agent: SmartologyBot
332User-agent: SMTBot
333User-agent: Sottopop
334User-agent: StatistikAustria
335User-agent: StatusNestBacklinkSpider
336User-agent: Storebot-Google
337User-agent: Stripebot
338User-agent: Taboola
339User-agent: Taboolabot
340User-agent: TechSpyBot
341User-agent: trendictionbot
342User-agent: TrendsmapResolver
343User-agent: TTD-Content
344User-agent: TurnitinBot
345User-agent: TweetmemeBot
346User-agent: Twingly
347User-agent: Twingly Recon-Sjostrom
348User-agent: uipbot
349User-agent: um-FC
350User-agent: um-IC
351User-agent: um-LN
352User-agent: UpContent
353User-agent: UpContent-QA
354User-agent: UpContent-Test
355User-agent: Verity
356User-agent: virustotal
357User-agent: W3 Validator Services
358User-agent: weborama-fetcher
359User-agent: webspidermount
360User-agent: WebTrackrCrawler
361User-agent: webzio
362User-agent: wknd-bot
363User-agent: woorankreview
364User-agent: XoviOnpageCrawler
365User-agent: YaDirectFetcher
366User-agent: Yahoo Ad monitoring
367User-agent: Yahoo Japan
368User-agent: YaK
369User-agent: Yext Inc
370User-agent: YextBot
371User-agent: Zoombot
372User-agent: ZoominfoBot
373Disallow: /search
374Disallow: /comments/
375Disallow: /_
376Disallow: /css
377Disallow: /fonts
378Disallow: /media
379Disallow: /piano
380Disallow: /static
381Crawl-delay: 10