NeuralCrawl

Vox / robots.txt

← back to vox.com · change detected 4d ago (2026-08-19T17:42:34Z) · modified +100 −0

What changed

Diff old (2026-06-20T01:10:30Z) → new (2026-08-19T17:42:34Z)

@@ -11,402 +11,502 @@
1111  User-agent: A6-Indexer
1212  Disallow: /
1313  Allow: /sp/
14+ Allow: /strategist
1415 
1516  User-agent: AhrefsBot
1617  Disallow: /
1718  Allow: /sp/
19+ Allow: /strategist
1820 
1921  User-agent: AI2Bot
2022  Disallow: /
2123  Allow: /sp/
24+ Allow: /strategist
2225 
2326  User-agent: AI2Bot-DeepResearchEval
2427  Disallow: /
2528  Allow: /sp/
29+ Allow: /strategist
2630 
2731  User-agent: Ai2Bot-Dolma
2832  Disallow: /
2933  Allow: /sp/
34+ Allow: /strategist
3035 
3136  User-agent: aiHitBot
3237  Disallow: /
3338  Allow: /sp/
39+ Allow: /strategist
3440 
3541  User-agent: Anchor Browser
3642  Disallow: /
3743  Allow: /sp/
44+ Allow: /strategist
3845 
3946  User-agent: anthropic-ai
4047  Disallow: /
4148  Allow: /sp/
49+ Allow: /strategist
4250 
4351  User-agent: ApifyBot
4452  Disallow: /
4553  Allow: /sp/
54+ Allow: /strategist
4655 
4756  User-agent: ApifyWebsiteContentCrawler
4857  Disallow: /
4958  Allow: /sp/
59+ Allow: /strategist
5060 
5161  User-agent: Applebot-Extended
5262  Disallow: /
5363  Allow: /sp/
64+ Allow: /strategist
5465 
5566  User-agent: Baiduspider
5667  Disallow: /
5768  Allow: /sp/
69+ Allow: /strategist
5870 
5971  User-agent: betaBot
6072  Disallow: /
6173  Allow: /sp/
74+ Allow: /strategist
6275 
6376  User-agent: BLEXBot
6477  Disallow: /
6578  Allow: /sp/
79+ Allow: /strategist
6680 
6781  User-agent: botify
6882  Disallow: /
6983  Allow: /sp/
84+ Allow: /strategist
7085 
7186  User-agent: brands-bot-logo
7287  Disallow: /
7388  Allow: /sp/
89+ Allow: /strategist
7490 
7591  User-agent: Brightbot
7692  Disallow: /
7793  Allow: /sp/
94+ Allow: /strategist
7895 
7996  User-agent: Bytespider
8097  Disallow: /
8198  Allow: /sp/
99+ Allow: /strategist
82100 
83101  User-agent: calculon
84102  Disallow: /
85103  Allow: /sp/
104+ Allow: /strategist
86105 
87106  User-agent: CCBot
88107  Disallow: /
89108  Allow: /sp/
109+ Allow: /strategist
90110 
91111  User-agent: ChatGLM-Spider
92112  Disallow: /
93113  Allow: /sp/
114+ Allow: /strategist
94115 
95116  User-agent: ChatGPT-User
96117  Disallow: /
97118  Allow: /sp/
119+ Allow: /strategist
98120 
99121  User-agent: Claude-SearchBot
100122  Disallow: /
101123  Allow: /sp/
124+ Allow: /strategist
102125 
103126  User-agent: Claude-User
104127  Disallow: /
105128  Allow: /sp/
129+ Allow: /strategist
106130 
107131  User-agent: Claude-Web
108132  Disallow: /
109133  Allow: /sp/
134+ Allow: /strategist
110135 
111136  User-agent: ClaudeBot
112137  Disallow: /
113138  Allow: /sp/
139+ Allow: /strategist
114140 
115141  User-agent: CloudVertexBot
116142  Disallow: /
117143  Allow: /sp/
144+ Allow: /strategist
118145 
119146  User-agent: cohere-ai
120147  Disallow: /
121148  Allow: /sp/
149+ Allow: /strategist
122150 
123151  User-agent: cohere-training-data-crawler
124152  Disallow: /
125153  Allow: /sp/
154+ Allow: /strategist
126155 
127156  User-agent: [email protected]
128157  Disallow: /
129158  Allow: /sp/
159+ Allow: /strategist
130160 
131161  User-agent: Cotoyogi
132162  Disallow: /
133163  Allow: /sp/
164+ Allow: /strategist
134165 
135166  User-agent: Crawl4AI
136167  Disallow: /
137168  Allow: /sp/
169+ Allow: /strategist
138170 
139171  User-agent: CrunchBot
140172  Disallow: /
141173  Allow: /sp/
174+ Allow: /strategist
142175 
143176  User-agent: deepcrawl
144177  Disallow: /
145178  Allow: /sp/
179+ Allow: /strategist
146180 
147181  User-agent: DeepSeekBot
148182  Disallow: /
149183  Allow: /sp/
184+ Allow: /strategist
150185 
151186  User-agent: Diffbot
152187  Disallow: /
153188  Allow: /sp/
189+ Allow: /strategist
154190 
155191  User-agent: DirBuster
156192  Disallow: /
157193  Allow: /sp/
194+ Allow: /strategist
158195 
159196  User-agent: DTS
160197  Disallow: /
161198  Allow: /sp/
199+ Allow: /strategist
162200 
163201  User-agent: DuckAssistBot
164202  Disallow: /
165203  Allow: /sp/
204+ Allow: /strategist
166205 
167206  User-agent: EasouSpider
168207  Disallow: /
169208  Allow: /sp/
209+ Allow: /strategist
170210 
171211  User-agent: FacebookBot
172212  Disallow: /
173213  Allow: /sp/
214+ Allow: /strategist
174215 
175216  User-agent: FirecrawlAgent
176217  Disallow: /
177218  Allow: /sp/
219+ Allow: /strategist
178220 
179221  User-agent: FriendlyCrawler
180222  Disallow: /
181223  Allow: /sp/
224+ Allow: /strategist
182225 
183226  User-agent: GlutenCrawler
184227  Disallow: /
185228  Allow: /sp/
229+ Allow: /strategist
186230 
187231  User-agent: Google-CloudVertexBot
188232  Disallow: /
189233  Allow: /sp/
234+ Allow: /strategist
190235 
191236  User-agent: Google-Extended
192237  Disallow: /
193238  Allow: /sp/
239+ Allow: /strategist
194240 
195241  User-agent: GrapeshotCrawler
196242  Disallow: /
197243  Allow: /sp/
244+ Allow: /strategist
198245 
199246  User-agent: iaskspider
200247  Disallow: /
201248  Allow: /sp/
249+ Allow: /strategist
202250 
203251  User-agent: ICC-Crawler
204252  Disallow: /
205253  Allow: /sp/
254+ Allow: /strategist
206255 
207256  User-agent: ImagesiftBot
208257  Disallow: /
209258  Allow: /sp/
259+ Allow: /strategist
210260 
211261  User-agent: img2dataset
212262  Disallow: /
213263  Allow: /sp/
264+ Allow: /strategist
214265 
215266  User-agent: JackCrawl
216267  Disallow: /
217268  Allow: /sp/
269+ Allow: /strategist
218270 
219271  User-agent: Jetslide
220272  Disallow: /
221273  Allow: /sp/
274+ Allow: /strategist
222275 
223276  User-agent: linkapedia_bot
224277  Disallow: /
225278  Allow: /sp/
279+ Allow: /strategist
226280 
227281  User-agent: Mediatoolkitbot
228282  Disallow: /
229283  Allow: /sp/
284+ Allow: /strategist
230285 
231286  User-agent: MegaIndex
232287  Disallow: /
233288  Allow: /sp/
289+ Allow: /strategist
234290 
235291  User-agent: meta-externalagent
236292  Disallow: /
237293  Allow: /sp/
294+ Allow: /strategist
238295 
239296  User-agent: meta-externalfetcher
240297  Disallow: /
241298  Allow: /sp/
299+ Allow: /strategist
242300 
243301  User-agent: Microsoft BITS
244302  Disallow: /
245303  Allow: /sp/
304+ Allow: /strategist
246305 
247306  User-agent: Microsoft URL Control
248307  Disallow: /
249308  Allow: /sp/
309+ Allow: /strategist
250310 
251311  User-agent: Microsoft-WebDAV-MiniRedir
252312  Disallow: /
253313  Allow: /sp/
314+ Allow: /strategist
254315 
255316  User-agent: MistralAI-User
256317  Disallow: /
257318  Allow: /sp/
319+ Allow: /strategist
258320 
259321  User-agent: news-please
260322  Disallow: /
261323  Allow: /sp/
324+ Allow: /strategist
262325 
263326  User-agent: Novellum
264327  Disallow: /
265328  Allow: /sp/
329+ Allow: /strategist
266330 
267331  User-agent: NTENTbot
268332  Disallow: /
269333  Allow: /sp/
334+ Allow: /strategist
270335 
271336  User-agent: Nutch
272337  Disallow: /
273338  Allow: /sp/
339+ Allow: /strategist
274340 
275341  User-agent: omgili
276342  Disallow: /
277343  Allow: /sp/
344+ Allow: /strategist
278345 
279346  User-agent: Omgilibot
280347  Disallow: /
281348  Allow: /sp/
349+ Allow: /strategist
282350 
283351  User-agent: OpenLinkProfiler
284352  Disallow: /
285353  Allow: /sp/
354+ Allow: /strategist
286355 
287356  User-agent: Perplexity-User
288357  Disallow: /
289358  Allow: /sp/
359+ Allow: /strategist
290360 
291361  User-agent: PerplexityBot
292362  Disallow: /
293363  Allow: /sp/
364+ Allow: /strategist
294365 
295366  User-agent: PetalBot
296367  Disallow: /
297368  Allow: /sp/
369+ Allow: /strategist
298370 
299371  User-agent: Primalbot
300372  Disallow: /
301373  Allow: /sp/
374+ Allow: /strategist
302375 
303376  User-agent: python-requests
304377  Disallow: /
305378  Allow: /sp/
379+ Allow: /strategist
306380 
307381  User-agent: Re-Animator
308382  Disallow: /
309383  Allow: /sp/
384+ Allow: /strategist
310385 
311386  User-agent: Riddler
312387  Disallow: /
313388  Allow: /sp/
389+ Allow: /strategist
314390 
315391  User-agent: SafeDNSBot
316392  Disallow: /
317393  Allow: /sp/
394+ Allow: /strategist
318395 
319396  User-agent: SBIntuitionsBot
320397  Disallow: /
321398  Allow: /sp/
399+ Allow: /strategist
322400 
323401  User-agent: Scrapy
324402  Disallow: /
325403  Allow: /sp/
404+ Allow: /strategist
326405 
327406  User-agent: SearchAtlas
328407  Disallow: /
329408  Allow: /sp/
409+ Allow: /strategist
330410 
331411  User-agent: SemanticScholarBot
332412  Disallow: /
333413  Allow: /sp/
414+ Allow: /strategist
334415 
335416  User-agent: SemrushBot
336417  Disallow: /
337418  Allow: /sp/
419+ Allow: /strategist
338420 
339421  User-agent: SentiBot
340422  Disallow: /
341423  Allow: /sp/
424+ Allow: /strategist
342425 
343426  User-agent: Sogou
344427  Disallow: /
345428  Allow: /sp/
429+ Allow: /strategist
346430 
347431  User-agent: Spider-Pig
348432  Disallow: /
349433  Allow: /sp/
434+ Allow: /strategist
350435 
351436  User-agent: sqlmap
352437  Disallow: /
353438  Allow: /sp/
439+ Allow: /strategist
354440 
355441  User-agent: StorygizeBot
356442  Disallow: /
357443  Allow: /sp/
444+ Allow: /strategist
358445 
359446  User-agent: Test Certificate Info
360447  Disallow: /
361448  Allow: /sp/
449+ Allow: /strategist
362450 
363451  User-agent: TikTokSpider
364452  Disallow: /
365453  Allow: /sp/
454+ Allow: /strategist
366455 
367456  User-agent: Timpibot
368457  Disallow: /
369458  Allow: /sp/
459+ Allow: /strategist
370460 
371461  User-agent: Turnitin
372462  Disallow: /
373463  Allow: /sp/
464+ Allow: /strategist
374465 
375466  User-agent: uipbot
376467  Disallow: /
377468  Allow: /sp/
469+ Allow: /strategist
378470 
379471  User-agent: vBSEO
380472  Disallow: /
381473  Allow: /sp/
474+ Allow: /strategist
382475 
383476  User-agent: VegeBot
384477  Disallow: /
385478  Allow: /sp/
479+ Allow: /strategist
386480 
387481  User-agent: VelenPublicWebCrawler
388482  Disallow: /
389483  Allow: /sp/
484+ Allow: /strategist
390485 
391486  User-agent: Web Link Validator
392487  Disallow: /
393488  Allow: /sp/
489+ Allow: /strategist
394490 
395491  User-agent: WebCruiser
396492  Disallow: /
397493  Allow: /sp/
494+ Allow: /strategist
398495 
399496  User-agent: webzio-extended
400497  Disallow: /
401498  Allow: /sp/
499+ Allow: /strategist
402500 
403501  User-agent: Xenu
404502  Disallow: /
405503  Allow: /sp/
504+ Allow: /strategist
406505 
407506  User-agent: YouBot
408507  Disallow: /
409508  Allow: /sp/
509+ Allow: /strategist
410510 
411511  User-agent: *
412512  Disallow: /wp-admin/