NeuralCrawl

Slate / robots.txt snapshot

← back to slate.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 8151 bytes · sha256 a7a6e062bc78a8ac · raw

final URL: https://slate.com/robots.txt

1User-agent: *
2Disallow: /search
3Disallow: /comments/
4Disallow: /_
5
6Sitemap: https://slate.com/sitemaps/sitemap-index.xml
7
8User-agent: AmazonAdBot
9User-agent: Amazon-Advertising-ad-standards-bot
10Allow: /
11
12# Known agents
13
14# AI Data Scraper, Scraper
15User-agent: 008
16User-agent: AdsTxtCrawler
17User-agent: Ai2Bot-Dolma
18User-agent: Amazonbot
19User-agent: Applebot-Extended
20User-agent: Bytespider
21User-agent: CCBot
22User-agent: ChatGLM-Spider
23User-agent: ClaudeBot
24User-agent: CloudVertexBot
25User-agent: cohere-training-data-crawler
26User-agent: Cotoyogi
27User-agent: Dataprovider.com
28User-agent: Datenbank Crawler
29User-agent: dcrawl
30User-agent: FacebookBot
31User-agent: Google-Extended
32User-agent: GoogleOther
33User-agent: GPTBot
34User-agent: HelloworkJobPostingBot
35User-agent: HTTrack
36User-agent: HTTrack 3.0
37User-agent: ICC-Crawler
38User-agent: imageSpider
39User-agent: IndeedJobBot
40User-agent: Kangaroo Bot
41User-agent: KrawlerBot
42User-agent: laion-huggingface-processor
43User-agent: LCC
44User-agent: meta-externalagent
45User-agent: MetaInspector
46User-agent: MetaJobBot
47User-agent: netEstate Imprint Crawler
48User-agent: newspaper
49User-agent: Nutch
50User-agent: nyt_scraping
51User-agent: Offline Explorer
52User-agent: omgili
53User-agent: OpenindexSpider
54User-agent: PanguBot
55User-agent: Potions
56User-agent: SBIntuitionsBot
57User-agent: Scrapy
58User-agent: ServerHunterSpider
59User-agent: Spider
60User-agent: StatsDroneBot
61User-agent: Timpibot
62User-agent: VelenPublicWebCrawler
63User-agent: webzio-extended
64Disallow: /
65
66# Intelligence Gatherer, SEO Crawler
67User-agent: A360-Search
68User-agent: aa
69User-agent: ActiveComply
70User-agent: Adagiobot
71User-agent: adbeat_bot
72User-agent: adidxbot
73User-agent: AdkernelTopicCrawler
74User-agent: admantx
75User-agent: AdsBot-Google
76User-agent: AdsBot-Google-Mobile
77User-agent: Ahrefs Site Audit
78User-agent: AhrefsBot
79User-agent: AhrefsSiteAudit
80User-agent: aiHitBot
81User-agent: Amazing-SearchBot
82User-agent: AmazingTest-SearchBot
83User-agent: Amazon AdBot
84User-agent: AmazonProductDiscovery
85User-agent: AndersPinkBot
86User-agent: ArchiveBot
87User-agent: Assaybot
88User-agent: Attracta
89User-agent: AudigentAdBot
90User-agent: Audisto Crawler
91User-agent: Automattic Analytics Crawler
92User-agent: Awario
93User-agent: AwarioBot
94User-agent: AwarioSmartBot
95User-agent: Barkrowler
96User-agent: BigUpDataBot
97User-agent: Bing Ads
98User-agent: BitSightBot
99User-agent: Blackboard
100User-agent: BLEXBot
101User-agent: BomboraBot
102User-agent: botify
103User-agent: BrandVerity
104User-agent: BrightEdge Bot
105User-agent: BrightEdge Crawler
106User-agent: Buck
107User-agent: CaliberBot
108User-agent: Caliperbot
109User-agent: channable
110User-agent: CheckMarkNetwork
111User-agent: Cincraw
112User-agent: ClarityBot
113User-agent: Clearscopebot
114User-agent: Clickagy Intelligence Bot
115User-agent: Clickagy Intelligence Bot v2
116User-agent: Cloudflare Custom Hostname Verification
117User-agent: Cloudflare Stream Webhook
118User-agent: Cloudflare-Healthchecks
119User-agent: CloudFlare-Prefetch
120User-agent: Cloudflare-SSLDetector
121User-agent: Cloudflare-Traffic-Manager
122User-agent: Cloudtrellis
123User-agent: cludo.com
124User-agent: Cocolyzebot
125User-agent: cognitiveSEO Crawler
126User-agent: contentking
127User-agent: ContextualBot
128User-agent: contxbot
129User-agent: Convermax
130User-agent: CriteoBot
131User-agent: Cxense
132User-agent: cXensebot
133User-agent: DataForSEO
134User-agent: DataForSEO Bot
135User-agent: DataForSeoBot
136User-agent: deepcrawl
137User-agent: deepcrawl.com
138User-agent: DomainStatsBot
139User-agent: DotBot
140User-agent: Dragonbot
141User-agent: ds9
142User-agent: DVbot
143User-agent: Echobot Bot
144User-agent: EcoVadisSustainabilityBot
145User-agent: EpivozCrawler
146User-agent: ev-crawler
147User-agent: EzLynx
148User-agent: EzoicBot
149User-agent: EzoicBot-Nicheiq
150User-agent: factset_spyderbot
151User-agent: FDL Stats Bot
152User-agent: Fedicabot
153User-agent: Google AdsBot
154User-agent: Google AdSense
155User-agent: Google Digital Asset Links
156User-agent: Google StoreBot
157User-agent: Google-Adwords-Instant
158User-agent: Google-Display-Ads-Bot
159User-agent: Google-Structured-Data-Testing
160User-agent: GrapeshotCrawler
161User-agent: GroovinaAdsbot
162User-agent: HubSpot
163User-agent: Huckabot
164User-agent: Huckabuy Bot
165User-agent: hypestat
166User-agent: IAS crawler
167User-agent: ias-ie
168User-agent: ImagesiftBot
169User-agent: Innguma
170User-agent: InteractiveAdvertisingBureau
171User-agent: IonCrawl
172User-agent: James BOT
173User-agent: Jugendschutzprogramm-Crawler
174User-agent: KargoBot
175User-agent: KargoBot-Artemis-Mobile
176User-agent: Klaviyo
177User-agent: KStandBot
178User-agent: Leikibot
179User-agent: LightspeedSystemsCrawler
180User-agent: LinkCheck by Siteimprove
181User-agent: LinkChecker Bot
182User-agent: linkchecker.pro
183User-agent: LinkCheckerBot
184User-agent: linkdexbot
185User-agent: linkfluence
186User-agent: LinksIndexerBot
187User-agent: LinkWalker
188User-agent: Macrobondbot
189User-agent: magpie-crawler
190User-agent: MarketGoo
191User-agent: MBCrawler
192User-agent: McontextualBot
193User-agent: MedialogiaBot
194User-agent: MediaMonitoringBot
195User-agent: Mediapartners-Google
196User-agent: Mediatoolkitbot
197User-agent: Mediavine Medatada Parser
198User-agent: MediavineMetadataParser
199User-agent: MegaIndex.ru
200User-agent: meta-externalads
201User-agent: MgidBot
202User-agent: Missinglettr Bot
203User-agent: MixrankBot
204User-agent: MJ12bot
205User-agent: ModernizeBot
206User-agent: Moz dotbot
207User-agent: Moz rogerbot
208User-agent: MuckRack
209User-agent: NanoInteractive
210User-agent: Netcraft
211User-agent: NetcraftSurveyAgent
212User-agent: netEstate NE Crawler
213User-agent: Neticle Crawler
214User-agent: NetSeer crawler
215User-agent: Netvibes
216User-agent: NewsRoom.BI
217User-agent: Nitro-
218User-agent: NitroBot
219User-agent: noorobot
220User-agent: NostoCrawlerBot
221User-agent: OAI-AdsBot
222User-agent: OnCrawl
223User-agent: OnCrawl Mobile
224User-agent: online-webceo-bot
225User-agent: outboundbuddy-enrichment
226User-agent: outbrain
227User-agent: Owler
228User-agent: page-preview-tool
229User-agent: Pandalytics
230User-agent: panscient.com
231User-agent: parse.ly scraper
232User-agent: PartnerOptimizer-bot
233User-agent: Prerender
234User-agent: Pro Sitemaps
235User-agent: Pro Sitemaps Generator
236User-agent: productsup.io/crawler
237User-agent: proximic
238User-agent: Quantcastbot
239User-agent: QuuuBot
240User-agent: Qwarrybot
241User-agent: Readable
242User-agent: RetroListeCOM
243User-agent: RevvimGort
244User-agent: rogerbot-crawler
245User-agent: RSiteAuditor
246User-agent: RyteBot
247User-agent: SalesViewerBot
248User-agent: scoop.it
249User-agent: Scope3
250User-agent: [email protected]
251User-agent: Screaming Frog SEO Spider
252User-agent: SearchAtlas Bot
253User-agent: SEBot-WA
254User-agent: SemrushBot
255User-agent: SemrushBot-BA
256User-agent: SemrushBot-CT
257User-agent: SemrushBot-OCOB
258User-agent: SemrushBot-SA
259User-agent: SemrushBot-SI
260User-agent: SemrushBotBacklinks
261User-agent: SentiBot
262User-agent: Senutobot
263User-agent: seo-audit-check-bot
264User-agent: seo4ajax
265User-agent: seo4ajax.com
266User-agent: SeobilityBot
267User-agent: SEOkicks
268User-agent: SEOlizer
269User-agent: SERankingBacklinksBot
270User-agent: serpstatbot
271User-agent: SiteAuditBot
272User-agent: Sitebulb
273User-agent: SiteCheck-sitecrawl
274User-agent: SiteCheckerBotCrawler
275User-agent: SiteGuruCrawler
276User-agent: Siteimprove Crawl
277User-agent: Siteliner
278User-agent: SlickBot
279User-agent: slickstream
280User-agent: SmartologyBot
281User-agent: SMTBot
282User-agent: StatistikAustria
283User-agent: StatusNestBacklinkSpider
284User-agent: Storebot-Google
285User-agent: Stripebot
286User-agent: Taboola
287User-agent: Taboolabot
288User-agent: trendictionbot
289User-agent: TrendsmapResolver
290User-agent: TTD-Content
291User-agent: TurnitinBot
292User-agent: TweetmemeBot
293User-agent: Twingly
294User-agent: Twingly Recon-Sjostrom
295User-agent: uipbot
296User-agent: um-FC
297User-agent: um-IC
298User-agent: um-LN
299User-agent: Verity
300User-agent: virustotal
301User-agent: W3 Validator Services
302User-agent: weborama-fetcher
303User-agent: webspidermount
304User-agent: WebTrackrCrawler
305User-agent: webzio
306User-agent: wknd-bot
307User-agent: woorankreview
308User-agent: XoviOnpageCrawler
309User-agent: YaDirectFetcher
310User-agent: Yahoo Ad monitoring
311User-agent: Yahoo Japan
312User-agent: YaK
313User-agent: Yext Inc
314User-agent: YextBot
315User-agent: Zoombot
316User-agent: ZoominfoBot
317Disallow: /search
318Disallow: /comments/
319Disallow: /_
320Disallow: /css
321Disallow: /fonts
322Disallow: /media
323Disallow: /piano
324Disallow: /static
325Crawl-delay: 10