NeuralCrawl

Slate / robots.txt snapshot

← back to slate.com · fetched 2026-09-09T00:14:16Z (10d ago) · HTTP 200 · 9454 bytes · sha256 b8ca0ee2a8beb806 · raw

final URL: https://slate.com/robots.txt

1User-agent: *
2Disallow: /search
3Disallow: /comments/
4Disallow: /_
5
6Sitemap: https://slate.com/sitemaps/sitemap-index.xml
7
8User-agent: AmazonAdBot
9User-agent: Amazon-Advertising-ad-standards-bot
10Allow: /
11
12# Known agents
13
14# AI Data Scraper, Scraper
15User-agent: 008
16User-agent: AdsTxtCrawler
17User-agent: AI2Bot
18User-agent: Ai2Bot-Dolma
19User-agent: Amazonbot
20User-agent: ApifyBot
21User-agent: ApifyWebsiteContentCrawler
22User-agent: Applebot-Extended
23User-agent: bedrockbot
24User-agent: Bravebot
25User-agent: Brightbot
26User-agent: Bytespider
27User-agent: CCBot
28User-agent: ChatGLM-Spider
29User-agent: ClaudeBot
30User-agent: CloudVertexBot
31User-agent: cohere-training-data-crawler
32User-agent: CohereBot
33User-agent: Cotoyogi
34User-agent: CragCrawler
35User-agent: Crawlspace
36User-agent: Dataprovider.com
37User-agent: Datenbank Crawler
38User-agent: dcrawl
39User-agent: DeepSeekBot
40User-agent: Diffbot
41User-agent: Doubaobot
42User-agent: ERNIEBot
43User-agent: ExaBot
44User-agent: FacebookBot
45User-agent: FirecrawlAgent
46User-agent: Google-Extended
47User-agent: GoogleOther
48User-agent: GPTBot
49User-agent: HelloworkJobPostingBot
50User-agent: HenkBot
51User-agent: HTTrack
52User-agent: HTTrack 3.0
53User-agent: ICC-Crawler
54User-agent: imageSpider
55User-agent: IndeedJobBot
56User-agent: Kangaroo Bot
57User-agent: KimiBot
58User-agent: KimiCrawler
59User-agent: KrawlerBot
60User-agent: laion-huggingface-processor
61User-agent: LCC
62User-agent: meta-externalagent
63User-agent: MetaInspector
64User-agent: MetaJobBot
65User-agent: micro-crawl
66User-agent: MistralBot
67User-agent: MoonshotBot
68User-agent: MoonSpider
69User-agent: Mozilla-Tabstack
70User-agent: netEstate Imprint Crawler
71User-agent: newspaper
72User-agent: Nutch
73User-agent: nyt_scraping
74User-agent: Offline Explorer
75User-agent: omgili
76User-agent: OpenindexSpider
77User-agent: PanguBot
78User-agent: Potions
79User-agent: Querit-SearchBot
80User-agent: QueritBot
81User-agent: QwenBot
82User-agent: Reflectionbot
83User-agent: SalamandraVLM
84User-agent: SBIntuitionsBot
85User-agent: Scrapy
86User-agent: ServerHunterSpider
87User-agent: ShapBot
88User-agent: Spider
89User-agent: StatsDroneBot
90User-agent: TavilyBot
91User-agent: Terra Cotta
92User-agent: TerraCotta
93User-agent: Thinkbot
94User-agent: Timpibot
95User-agent: VelenPublicWebCrawler
96User-agent: webzio-extended
97User-agent: YouBot
98Disallow: /
99
100# Intelligence Gatherer, SEO Crawler
101User-agent: A360-Search
102User-agent: aa
103User-agent: ActiveComply
104User-agent: Adagiobot
105User-agent: adbeat_bot
106User-agent: adidxbot
107User-agent: AdkernelTopicCrawler
108User-agent: admantx
109User-agent: AdmixerBot
110User-agent: AdsBot-Google
111User-agent: AdsBot-Google-Mobile
112User-agent: AffsignalCrawler
113User-agent: Ahrefs Site Audit
114User-agent: AhrefsBot
115User-agent: AhrefsSiteAudit
116User-agent: aiHitBot
117User-agent: Amazing-SearchBot
118User-agent: AmazingTest-SearchBot
119User-agent: Amazon AdBot
120User-agent: AmazonProductDiscovery
121User-agent: AndersPinkBot
122User-agent: ArchiveBot
123User-agent: Assaybot
124User-agent: Attracta
125User-agent: AudigentAdBot
126User-agent: Audisto Crawler
127User-agent: Automattic Analytics Crawler
128User-agent: Awario
129User-agent: AwarioBot
130User-agent: AwarioSmartBot
131User-agent: Barkrowler
132User-agent: BigUpDataBot
133User-agent: Bing Ads
134User-agent: BitSightBot
135User-agent: Blackboard
136User-agent: BLEXBot
137User-agent: BomboraBot
138User-agent: botify
139User-agent: BrandVerity
140User-agent: BrightEdge Bot
141User-agent: BrightEdge Crawler
142User-agent: Buck
143User-agent: CaliberBot
144User-agent: Caliperbot
145User-agent: captify-crawler
146User-agent: channable
147User-agent: CheckMarkNetwork
148User-agent: Cincraw
149User-agent: ClarityBot
150User-agent: Clearscopebot
151User-agent: Clickagy Intelligence Bot
152User-agent: Clickagy Intelligence Bot v2
153User-agent: Cloudflare Custom Hostname Verification
154User-agent: Cloudflare Stream Webhook
155User-agent: Cloudflare-Healthchecks
156User-agent: CloudFlare-Prefetch
157User-agent: Cloudflare-SSLDetector
158User-agent: Cloudflare-Traffic-Manager
159User-agent: Cloudtrellis
160User-agent: cludo.com
161User-agent: Cocolyzebot
162User-agent: cognitiveSEO Crawler
163User-agent: contentking
164User-agent: ContextualBot
165User-agent: contxbot
166User-agent: Convermax
167User-agent: CriteoBot
168User-agent: Cxense
169User-agent: cXensebot
170User-agent: DataForSEO
171User-agent: DataForSEO Bot
172User-agent: DataForSeoBot
173User-agent: deepcrawl
174User-agent: deepcrawl.com
175User-agent: DomainStatsBot
176User-agent: DotBot
177User-agent: Dragonbot
178User-agent: Dragonfly
179User-agent: ds9
180User-agent: DVbot
181User-agent: Echobot Bot
182User-agent: EcoVadisSustainabilityBot
183User-agent: EpivozCrawler
184User-agent: ev-crawler
185User-agent: EzLynx
186User-agent: EzoicBot
187User-agent: EzoicBot-Nicheiq
188User-agent: factset_spyderbot
189User-agent: FDL Stats Bot
190User-agent: Fedicabot
191User-agent: GnowitNewsbot
192User-agent: Google AdsBot
193User-agent: Google AdSense
194User-agent: Google Digital Asset Links
195User-agent: Google StoreBot
196User-agent: Google-adstxt
197User-agent: Google-Adwords-Instant
198User-agent: Google-Display-Ads-Bot
199User-agent: Google-Structured-Data-Testing
200User-agent: GrapeshotCrawler
201User-agent: GroovinaAdsbot
202User-agent: HubSpot
203User-agent: Huckabot
204User-agent: Huckabuy Bot
205User-agent: hypestat
206User-agent: IAS crawler
207User-agent: ias-ie
208User-agent: IDG/IT
209User-agent: ImagesiftBot
210User-agent: Innguma
211User-agent: InteractiveAdvertisingBureau
212User-agent: IonCrawl
213User-agent: James BOT
214User-agent: Jugendschutzprogramm-Crawler
215User-agent: KargoBot
216User-agent: KargoBot-Artemis-Mobile
217User-agent: Klaviyo
218User-agent: KStandBot
219User-agent: Leikibot
220User-agent: LibreCrawl
221User-agent: LightspeedSystemsCrawler
222User-agent: LinkCheck by Siteimprove
223User-agent: LinkChecker Bot
224User-agent: linkchecker.pro
225User-agent: LinkCheckerBot
226User-agent: linkdexbot
227User-agent: linkfluence
228User-agent: LinksIndexerBot
229User-agent: LinkWalker
230User-agent: Macrobondbot
231User-agent: magpie-crawler
232User-agent: Mantisbot
233User-agent: MarketGoo
234User-agent: MBCrawler
235User-agent: McontextualBot
236User-agent: mediaboardbot
237User-agent: MedialogiaBot
238User-agent: MediaMonitoringBot
239User-agent: Mediapartners-Google
240User-agent: Mediatoolkitbot
241User-agent: Mediavine Medatada Parser
242User-agent: MediavineMetadataParser
243User-agent: MegaIndex.ru
244User-agent: meta-externalads
245User-agent: MgidBot
246User-agent: Missinglettr Bot
247User-agent: MixrankBot
248User-agent: MJ12bot
249User-agent: ModernizeBot
250User-agent: Moz dotbot
251User-agent: Moz rogerbot
252User-agent: MuckRack
253User-agent: NanoInteractive
254User-agent: Netcraft
255User-agent: NetcraftSurveyAgent
256User-agent: netEstate NE Crawler
257User-agent: Neticle Crawler
258User-agent: NetSeer crawler
259User-agent: Netvibes
260User-agent: NeuwoBot
261User-agent: NewsRoom.BI
262User-agent: Nitro-
263User-agent: NitroBot
264User-agent: noorobot
265User-agent: NostoCrawlerBot
266User-agent: OAI-AdsBot
267User-agent: OnCrawl
268User-agent: OnCrawl Mobile
269User-agent: online-webceo-bot
270User-agent: outboundbuddy-enrichment
271User-agent: outbrain
272User-agent: Owler
273User-agent: page-preview-tool
274User-agent: Pandalytics
275User-agent: panscient.com
276User-agent: parse.ly scraper
277User-agent: PartnerOptimizer-bot
278User-agent: peer39_crawler
279User-agent: Pro Sitemaps
280User-agent: Pro Sitemaps Generator
281User-agent: productsup.io/crawler
282User-agent: proximic
283User-agent: Quantcastbot
284User-agent: QuuuBot
285User-agent: Qwarrybot
286User-agent: RankMath Link Checker
287User-agent: RankurBot
288User-agent: RavenCrawler
289User-agent: Readable
290User-agent: RetroListeCOM
291User-agent: RevvimGort
292User-agent: rogerbot-crawler
293User-agent: RSiteAuditor
294User-agent: RyteBot
295User-agent: SalesViewerBot
296User-agent: scoop.it
297User-agent: Scope3
298User-agent: [email protected]
299User-agent: Screaming Frog SEO Spider
300User-agent: SearchAtlas Bot
301User-agent: SEBot-WA
302User-agent: SemrushBot
303User-agent: SemrushBot-BA
304User-agent: SemrushBot-CT
305User-agent: SemrushBot-OCOB
306User-agent: SemrushBot-SA
307User-agent: SemrushBot-SI
308User-agent: SemrushBotBacklinks
309User-agent: SentiBot
310User-agent: Senutobot
311User-agent: seo-audit-check-bot
312User-agent: seo4ajax
313User-agent: seo4ajax.com
314User-agent: SeobilityBot
315User-agent: SEOkicks
316User-agent: SEOlizer
317User-agent: SERankingBacklinksBot
318User-agent: SerpMinerBot
319User-agent: serpstatbot
320User-agent: SirdataBot
321User-agent: SiteAuditBot
322User-agent: Sitebulb
323User-agent: SiteCheck-sitecrawl
324User-agent: SiteCheckerBotCrawler
325User-agent: SiteGuruCrawler
326User-agent: Siteimprove Crawl
327User-agent: Siteliner
328User-agent: SlickBot
329User-agent: slickstream
330User-agent: SmartologyBot
331User-agent: SMTBot
332User-agent: Sottopop
333User-agent: StatistikAustria
334User-agent: StatusNestBacklinkSpider
335User-agent: Storebot-Google
336User-agent: Stripebot
337User-agent: Taboola
338User-agent: Taboolabot
339User-agent: TechSpyBot
340User-agent: trendictionbot
341User-agent: TrendsmapResolver
342User-agent: TTD-Content
343User-agent: TurnitinBot
344User-agent: TweetmemeBot
345User-agent: Twingly
346User-agent: Twingly Recon-Sjostrom
347User-agent: uipbot
348User-agent: um-FC
349User-agent: um-IC
350User-agent: um-LN
351User-agent: UpContent
352User-agent: UpContent-QA
353User-agent: UpContent-Test
354User-agent: Verity
355User-agent: virustotal
356User-agent: W3 Validator Services
357User-agent: weborama-fetcher
358User-agent: webspidermount
359User-agent: WebTrackrCrawler
360User-agent: webzio
361User-agent: wknd-bot
362User-agent: woorankreview
363User-agent: XoviOnpageCrawler
364User-agent: YaDirectFetcher
365User-agent: Yahoo Ad monitoring
366User-agent: Yahoo Japan
367User-agent: YaK
368User-agent: Yext Inc
369User-agent: YextBot
370User-agent: Zoombot
371User-agent: ZoominfoBot
372Disallow: /search
373Disallow: /comments/
374Disallow: /_
375Disallow: /css
376Disallow: /fonts
377Disallow: /media
378Disallow: /piano
379Disallow: /static
380Crawl-delay: 10