NeuralCrawl

Vox / robots.txt

← back to vox.com · change detected 4d ago (2026-08-19T23:50:02Z) · modified +0 −100

What changed

Diff old (2026-08-19T17:42:34Z) → new (2026-08-19T23:50:02Z)

@@ -11,502 +11,402 @@
1111  User-agent: A6-Indexer
1212  Disallow: /
1313  Allow: /sp/
14- Allow: /strategist
1514 
1615  User-agent: AhrefsBot
1716  Disallow: /
1817  Allow: /sp/
19- Allow: /strategist
2018 
2119  User-agent: AI2Bot
2220  Disallow: /
2321  Allow: /sp/
24- Allow: /strategist
2522 
2623  User-agent: AI2Bot-DeepResearchEval
2724  Disallow: /
2825  Allow: /sp/
29- Allow: /strategist
3026 
3127  User-agent: Ai2Bot-Dolma
3228  Disallow: /
3329  Allow: /sp/
34- Allow: /strategist
3530 
3631  User-agent: aiHitBot
3732  Disallow: /
3833  Allow: /sp/
39- Allow: /strategist
4034 
4135  User-agent: Anchor Browser
4236  Disallow: /
4337  Allow: /sp/
44- Allow: /strategist
4538 
4639  User-agent: anthropic-ai
4740  Disallow: /
4841  Allow: /sp/
49- Allow: /strategist
5042 
5143  User-agent: ApifyBot
5244  Disallow: /
5345  Allow: /sp/
54- Allow: /strategist
5546 
5647  User-agent: ApifyWebsiteContentCrawler
5748  Disallow: /
5849  Allow: /sp/
59- Allow: /strategist
6050 
6151  User-agent: Applebot-Extended
6252  Disallow: /
6353  Allow: /sp/
64- Allow: /strategist
6554 
6655  User-agent: Baiduspider
6756  Disallow: /
6857  Allow: /sp/
69- Allow: /strategist
7058 
7159  User-agent: betaBot
7260  Disallow: /
7361  Allow: /sp/
74- Allow: /strategist
7562 
7663  User-agent: BLEXBot
7764  Disallow: /
7865  Allow: /sp/
79- Allow: /strategist
8066 
8167  User-agent: botify
8268  Disallow: /
8369  Allow: /sp/
84- Allow: /strategist
8570 
8671  User-agent: brands-bot-logo
8772  Disallow: /
8873  Allow: /sp/
89- Allow: /strategist
9074 
9175  User-agent: Brightbot
9276  Disallow: /
9377  Allow: /sp/
94- Allow: /strategist
9578 
9679  User-agent: Bytespider
9780  Disallow: /
9881  Allow: /sp/
99- Allow: /strategist
10082 
10183  User-agent: calculon
10284  Disallow: /
10385  Allow: /sp/
104- Allow: /strategist
10586 
10687  User-agent: CCBot
10788  Disallow: /
10889  Allow: /sp/
109- Allow: /strategist
11090 
11191  User-agent: ChatGLM-Spider
11292  Disallow: /
11393  Allow: /sp/
114- Allow: /strategist
11594 
11695  User-agent: ChatGPT-User
11796  Disallow: /
11897  Allow: /sp/
119- Allow: /strategist
12098 
12199  User-agent: Claude-SearchBot
122100  Disallow: /
123101  Allow: /sp/
124- Allow: /strategist
125102 
126103  User-agent: Claude-User
127104  Disallow: /
128105  Allow: /sp/
129- Allow: /strategist
130106 
131107  User-agent: Claude-Web
132108  Disallow: /
133109  Allow: /sp/
134- Allow: /strategist
135110 
136111  User-agent: ClaudeBot
137112  Disallow: /
138113  Allow: /sp/
139- Allow: /strategist
140114 
141115  User-agent: CloudVertexBot
142116  Disallow: /
143117  Allow: /sp/
144- Allow: /strategist
145118 
146119  User-agent: cohere-ai
147120  Disallow: /
148121  Allow: /sp/
149- Allow: /strategist
150122 
151123  User-agent: cohere-training-data-crawler
152124  Disallow: /
153125  Allow: /sp/
154- Allow: /strategist
155126 
156127  User-agent: [email protected]
157128  Disallow: /
158129  Allow: /sp/
159- Allow: /strategist
160130 
161131  User-agent: Cotoyogi
162132  Disallow: /
163133  Allow: /sp/
164- Allow: /strategist
165134 
166135  User-agent: Crawl4AI
167136  Disallow: /
168137  Allow: /sp/
169- Allow: /strategist
170138 
171139  User-agent: CrunchBot
172140  Disallow: /
173141  Allow: /sp/
174- Allow: /strategist
175142 
176143  User-agent: deepcrawl
177144  Disallow: /
178145  Allow: /sp/
179- Allow: /strategist
180146 
181147  User-agent: DeepSeekBot
182148  Disallow: /
183149  Allow: /sp/
184- Allow: /strategist
185150 
186151  User-agent: Diffbot
187152  Disallow: /
188153  Allow: /sp/
189- Allow: /strategist
190154 
191155  User-agent: DirBuster
192156  Disallow: /
193157  Allow: /sp/
194- Allow: /strategist
195158 
196159  User-agent: DTS
197160  Disallow: /
198161  Allow: /sp/
199- Allow: /strategist
200162 
201163  User-agent: DuckAssistBot
202164  Disallow: /
203165  Allow: /sp/
204- Allow: /strategist
205166 
206167  User-agent: EasouSpider
207168  Disallow: /
208169  Allow: /sp/
209- Allow: /strategist
210170 
211171  User-agent: FacebookBot
212172  Disallow: /
213173  Allow: /sp/
214- Allow: /strategist
215174 
216175  User-agent: FirecrawlAgent
217176  Disallow: /
218177  Allow: /sp/
219- Allow: /strategist
220178 
221179  User-agent: FriendlyCrawler
222180  Disallow: /
223181  Allow: /sp/
224- Allow: /strategist
225182 
226183  User-agent: GlutenCrawler
227184  Disallow: /
228185  Allow: /sp/
229- Allow: /strategist
230186 
231187  User-agent: Google-CloudVertexBot
232188  Disallow: /
233189  Allow: /sp/
234- Allow: /strategist
235190 
236191  User-agent: Google-Extended
237192  Disallow: /
238193  Allow: /sp/
239- Allow: /strategist
240194 
241195  User-agent: GrapeshotCrawler
242196  Disallow: /
243197  Allow: /sp/
244- Allow: /strategist
245198 
246199  User-agent: iaskspider
247200  Disallow: /
248201  Allow: /sp/
249- Allow: /strategist
250202 
251203  User-agent: ICC-Crawler
252204  Disallow: /
253205  Allow: /sp/
254- Allow: /strategist
255206 
256207  User-agent: ImagesiftBot
257208  Disallow: /
258209  Allow: /sp/
259- Allow: /strategist
260210 
261211  User-agent: img2dataset
262212  Disallow: /
263213  Allow: /sp/
264- Allow: /strategist
265214 
266215  User-agent: JackCrawl
267216  Disallow: /
268217  Allow: /sp/
269- Allow: /strategist
270218 
271219  User-agent: Jetslide
272220  Disallow: /
273221  Allow: /sp/
274- Allow: /strategist
275222 
276223  User-agent: linkapedia_bot
277224  Disallow: /
278225  Allow: /sp/
279- Allow: /strategist
280226 
281227  User-agent: Mediatoolkitbot
282228  Disallow: /
283229  Allow: /sp/
284- Allow: /strategist
285230 
286231  User-agent: MegaIndex
287232  Disallow: /
288233  Allow: /sp/
289- Allow: /strategist
290234 
291235  User-agent: meta-externalagent
292236  Disallow: /
293237  Allow: /sp/
294- Allow: /strategist
295238 
296239  User-agent: meta-externalfetcher
297240  Disallow: /
298241  Allow: /sp/
299- Allow: /strategist
300242 
301243  User-agent: Microsoft BITS
302244  Disallow: /
303245  Allow: /sp/
304- Allow: /strategist
305246 
306247  User-agent: Microsoft URL Control
307248  Disallow: /
308249  Allow: /sp/
309- Allow: /strategist
310250 
311251  User-agent: Microsoft-WebDAV-MiniRedir
312252  Disallow: /
313253  Allow: /sp/
314- Allow: /strategist
315254 
316255  User-agent: MistralAI-User
317256  Disallow: /
318257  Allow: /sp/
319- Allow: /strategist
320258 
321259  User-agent: news-please
322260  Disallow: /
323261  Allow: /sp/
324- Allow: /strategist
325262 
326263  User-agent: Novellum
327264  Disallow: /
328265  Allow: /sp/
329- Allow: /strategist
330266 
331267  User-agent: NTENTbot
332268  Disallow: /
333269  Allow: /sp/
334- Allow: /strategist
335270 
336271  User-agent: Nutch
337272  Disallow: /
338273  Allow: /sp/
339- Allow: /strategist
340274 
341275  User-agent: omgili
342276  Disallow: /
343277  Allow: /sp/
344- Allow: /strategist
345278 
346279  User-agent: Omgilibot
347280  Disallow: /
348281  Allow: /sp/
349- Allow: /strategist
350282 
351283  User-agent: OpenLinkProfiler
352284  Disallow: /
353285  Allow: /sp/
354- Allow: /strategist
355286 
356287  User-agent: Perplexity-User
357288  Disallow: /
358289  Allow: /sp/
359- Allow: /strategist
360290 
361291  User-agent: PerplexityBot
362292  Disallow: /
363293  Allow: /sp/
364- Allow: /strategist
365294 
366295  User-agent: PetalBot
367296  Disallow: /
368297  Allow: /sp/
369- Allow: /strategist
370298 
371299  User-agent: Primalbot
372300  Disallow: /
373301  Allow: /sp/
374- Allow: /strategist
375302 
376303  User-agent: python-requests
377304  Disallow: /
378305  Allow: /sp/
379- Allow: /strategist
380306 
381307  User-agent: Re-Animator
382308  Disallow: /
383309  Allow: /sp/
384- Allow: /strategist
385310 
386311  User-agent: Riddler
387312  Disallow: /
388313  Allow: /sp/
389- Allow: /strategist
390314 
391315  User-agent: SafeDNSBot
392316  Disallow: /
393317  Allow: /sp/
394- Allow: /strategist
395318 
396319  User-agent: SBIntuitionsBot
397320  Disallow: /
398321  Allow: /sp/
399- Allow: /strategist
400322 
401323  User-agent: Scrapy
402324  Disallow: /
403325  Allow: /sp/
404- Allow: /strategist
405326 
406327  User-agent: SearchAtlas
407328  Disallow: /
408329  Allow: /sp/
409- Allow: /strategist
410330 
411331  User-agent: SemanticScholarBot
412332  Disallow: /
413333  Allow: /sp/
414- Allow: /strategist
415334 
416335  User-agent: SemrushBot
417336  Disallow: /
418337  Allow: /sp/
419- Allow: /strategist
420338 
421339  User-agent: SentiBot
422340  Disallow: /
423341  Allow: /sp/
424- Allow: /strategist
425342 
426343  User-agent: Sogou
427344  Disallow: /
428345  Allow: /sp/
429- Allow: /strategist
430346 
431347  User-agent: Spider-Pig
432348  Disallow: /
433349  Allow: /sp/
434- Allow: /strategist
435350 
436351  User-agent: sqlmap
437352  Disallow: /
438353  Allow: /sp/
439- Allow: /strategist
440354 
441355  User-agent: StorygizeBot
442356  Disallow: /
443357  Allow: /sp/
444- Allow: /strategist
445358 
446359  User-agent: Test Certificate Info
447360  Disallow: /
448361  Allow: /sp/
449- Allow: /strategist
450362 
451363  User-agent: TikTokSpider
452364  Disallow: /
453365  Allow: /sp/
454- Allow: /strategist
455366 
456367  User-agent: Timpibot
457368  Disallow: /
458369  Allow: /sp/
459- Allow: /strategist
460370 
461371  User-agent: Turnitin
462372  Disallow: /
463373  Allow: /sp/
464- Allow: /strategist
465374 
466375  User-agent: uipbot
467376  Disallow: /
468377  Allow: /sp/
469- Allow: /strategist
470378 
471379  User-agent: vBSEO
472380  Disallow: /
473381  Allow: /sp/
474- Allow: /strategist
475382 
476383  User-agent: VegeBot
477384  Disallow: /
478385  Allow: /sp/
479- Allow: /strategist
480386 
481387  User-agent: VelenPublicWebCrawler
482388  Disallow: /
483389  Allow: /sp/
484- Allow: /strategist
485390 
486391  User-agent: Web Link Validator
487392  Disallow: /
488393  Allow: /sp/
489- Allow: /strategist
490394 
491395  User-agent: WebCruiser
492396  Disallow: /
493397  Allow: /sp/
494- Allow: /strategist
495398 
496399  User-agent: webzio-extended
497400  Disallow: /
498401  Allow: /sp/
499- Allow: /strategist
500402 
501403  User-agent: Xenu
502404  Disallow: /
503405  Allow: /sp/
504- Allow: /strategist
505406 
506407  User-agent: YouBot
507408  Disallow: /
508409  Allow: /sp/
509- Allow: /strategist
510410 
511411  User-agent: *
512412  Disallow: /wp-admin/