NeuralCrawl

The Verge / robots.txt

← back to theverge.com · change detected 1d ago (2026-09-17T15:11:14Z) · modified +100 −100

What changed

Diff old (2026-06-20T01:10:30Z) → new (2026-09-17T15:11:14Z)

@@ -9,404 +9,404 @@
99  Allow: /
1010 
1111  User-agent: A6-Indexer
12+ Allow: /sp/
1213  Disallow: /
13- Allow: /sp/
1414 
1515  User-agent: AhrefsBot
16+ Allow: /sp/
1617  Disallow: /
17- Allow: /sp/
1818 
1919  User-agent: AI2Bot
20+ Allow: /sp/
2021  Disallow: /
21- Allow: /sp/
2222 
2323  User-agent: AI2Bot-DeepResearchEval
24+ Allow: /sp/
2425  Disallow: /
25- Allow: /sp/
2626 
2727  User-agent: Ai2Bot-Dolma
28+ Allow: /sp/
2829  Disallow: /
29- Allow: /sp/
3030 
3131  User-agent: aiHitBot
32+ Allow: /sp/
3233  Disallow: /
33- Allow: /sp/
3434 
3535  User-agent: Anchor Browser
36+ Allow: /sp/
3637  Disallow: /
37- Allow: /sp/
3838 
3939  User-agent: anthropic-ai
40+ Allow: /sp/
4041  Disallow: /
41- Allow: /sp/
4242 
4343  User-agent: ApifyBot
44+ Allow: /sp/
4445  Disallow: /
45- Allow: /sp/
4646 
4747  User-agent: ApifyWebsiteContentCrawler
48+ Allow: /sp/
4849  Disallow: /
49- Allow: /sp/
5050 
5151  User-agent: Applebot-Extended
52+ Allow: /sp/
5253  Disallow: /
53- Allow: /sp/
5454 
5555  User-agent: Baiduspider
56+ Allow: /sp/
5657  Disallow: /
57- Allow: /sp/
5858 
5959  User-agent: betaBot
60+ Allow: /sp/
6061  Disallow: /
61- Allow: /sp/
6262 
6363  User-agent: BLEXBot
64+ Allow: /sp/
6465  Disallow: /
65- Allow: /sp/
6666 
6767  User-agent: botify
68+ Allow: /sp/
6869  Disallow: /
69- Allow: /sp/
7070 
7171  User-agent: brands-bot-logo
72+ Allow: /sp/
7273  Disallow: /
73- Allow: /sp/
7474 
7575  User-agent: Brightbot
76+ Allow: /sp/
7677  Disallow: /
77- Allow: /sp/
7878 
7979  User-agent: Bytespider
80+ Allow: /sp/
8081  Disallow: /
81- Allow: /sp/
8282 
8383  User-agent: calculon
84+ Allow: /sp/
8485  Disallow: /
85- Allow: /sp/
8686 
8787  User-agent: CCBot
88+ Allow: /sp/
8889  Disallow: /
89- Allow: /sp/
9090 
9191  User-agent: ChatGLM-Spider
92+ Allow: /sp/
9293  Disallow: /
93- Allow: /sp/
9494 
9595  User-agent: ChatGPT-User
96+ Allow: /sp/
9697  Disallow: /
97- Allow: /sp/
9898 
9999  User-agent: Claude-SearchBot
100+ Allow: /sp/
100101  Disallow: /
101- Allow: /sp/
102102 
103103  User-agent: Claude-User
104+ Allow: /sp/
104105  Disallow: /
105- Allow: /sp/
106106 
107107  User-agent: Claude-Web
108+ Allow: /sp/
108109  Disallow: /
109- Allow: /sp/
110110 
111111  User-agent: ClaudeBot
112+ Allow: /sp/
112113  Disallow: /
113- Allow: /sp/
114114 
115115  User-agent: CloudVertexBot
116+ Allow: /sp/
116117  Disallow: /
117- Allow: /sp/
118118 
119119  User-agent: cohere-ai
120+ Allow: /sp/
120121  Disallow: /
121- Allow: /sp/
122122 
123123  User-agent: cohere-training-data-crawler
124+ Allow: /sp/
124125  Disallow: /
125- Allow: /sp/
126126 
127127  User-agent: [email protected]
128+ Allow: /sp/
128129  Disallow: /
129- Allow: /sp/
130130 
131131  User-agent: Cotoyogi
132+ Allow: /sp/
132133  Disallow: /
133- Allow: /sp/
134134 
135135  User-agent: Crawl4AI
136+ Allow: /sp/
136137  Disallow: /
137- Allow: /sp/
138138 
139139  User-agent: CrunchBot
140+ Allow: /sp/
140141  Disallow: /
141- Allow: /sp/
142142 
143143  User-agent: deepcrawl
144+ Allow: /sp/
144145  Disallow: /
145- Allow: /sp/
146146 
147147  User-agent: DeepSeekBot
148+ Allow: /sp/
148149  Disallow: /
149- Allow: /sp/
150150 
151151  User-agent: Diffbot
152+ Allow: /sp/
152153  Disallow: /
153- Allow: /sp/
154154 
155155  User-agent: DirBuster
156+ Allow: /sp/
156157  Disallow: /
157- Allow: /sp/
158158 
159159  User-agent: DTS
160+ Allow: /sp/
160161  Disallow: /
161- Allow: /sp/
162162 
163163  User-agent: DuckAssistBot
164+ Allow: /sp/
164165  Disallow: /
165- Allow: /sp/
166166 
167167  User-agent: EasouSpider
168+ Allow: /sp/
168169  Disallow: /
169- Allow: /sp/
170170 
171171  User-agent: FacebookBot
172+ Allow: /sp/
172173  Disallow: /
173- Allow: /sp/
174174 
175175  User-agent: FirecrawlAgent
176+ Allow: /sp/
176177  Disallow: /
177- Allow: /sp/
178178 
179179  User-agent: FriendlyCrawler
180+ Allow: /sp/
180181  Disallow: /
181- Allow: /sp/
182182 
183183  User-agent: GlutenCrawler
184+ Allow: /sp/
184185  Disallow: /
185- Allow: /sp/
186186 
187187  User-agent: Google-CloudVertexBot
188+ Allow: /sp/
188189  Disallow: /
189- Allow: /sp/
190190 
191191  User-agent: Google-Extended
192+ Allow: /sp/
192193  Disallow: /
193- Allow: /sp/
194194 
195195  User-agent: GrapeshotCrawler
196+ Allow: /sp/
196197  Disallow: /
197- Allow: /sp/
198198 
199199  User-agent: iaskspider
200+ Allow: /sp/
200201  Disallow: /
201- Allow: /sp/
202202 
203203  User-agent: ICC-Crawler
204+ Allow: /sp/
204205  Disallow: /
205- Allow: /sp/
206206 
207207  User-agent: ImagesiftBot
208+ Allow: /sp/
208209  Disallow: /
209- Allow: /sp/
210210 
211211  User-agent: img2dataset
212+ Allow: /sp/
212213  Disallow: /
213- Allow: /sp/
214214 
215215  User-agent: JackCrawl
216+ Allow: /sp/
216217  Disallow: /
217- Allow: /sp/
218218 
219219  User-agent: Jetslide
220+ Allow: /sp/
220221  Disallow: /
221- Allow: /sp/
222222 
223223  User-agent: linkapedia_bot
224+ Allow: /sp/
224225  Disallow: /
225- Allow: /sp/
226226 
227227  User-agent: Mediatoolkitbot
228+ Allow: /sp/
228229  Disallow: /
229- Allow: /sp/
230230 
231231  User-agent: MegaIndex
232+ Allow: /sp/
232233  Disallow: /
233- Allow: /sp/
234234 
235235  User-agent: meta-externalagent
236+ Allow: /sp/
236237  Disallow: /
237- Allow: /sp/
238238 
239239  User-agent: meta-externalfetcher
240+ Allow: /sp/
240241  Disallow: /
241- Allow: /sp/
242242 
243243  User-agent: Microsoft BITS
244+ Allow: /sp/
244245  Disallow: /
245- Allow: /sp/
246246 
247247  User-agent: Microsoft URL Control
248+ Allow: /sp/
248249  Disallow: /
249- Allow: /sp/
250250 
251251  User-agent: Microsoft-WebDAV-MiniRedir
252+ Allow: /sp/
252253  Disallow: /
253- Allow: /sp/
254254 
255255  User-agent: MistralAI-User
256+ Allow: /sp/
256257  Disallow: /
257- Allow: /sp/
258258 
259259  User-agent: news-please
260+ Allow: /sp/
260261  Disallow: /
261- Allow: /sp/
262262 
263263  User-agent: Novellum
264+ Allow: /sp/
264265  Disallow: /
265- Allow: /sp/
266266 
267267  User-agent: NTENTbot
268+ Allow: /sp/
268269  Disallow: /
269- Allow: /sp/
270270 
271271  User-agent: Nutch
272+ Allow: /sp/
272273  Disallow: /
273- Allow: /sp/
274274 
275275  User-agent: omgili
276+ Allow: /sp/
276277  Disallow: /
277- Allow: /sp/
278278 
279279  User-agent: Omgilibot
280+ Allow: /sp/
280281  Disallow: /
281- Allow: /sp/
282282 
283283  User-agent: OpenLinkProfiler
284+ Allow: /sp/
284285  Disallow: /
285- Allow: /sp/
286286 
287287  User-agent: Perplexity-User
288+ Allow: /sp/
288289  Disallow: /
289- Allow: /sp/
290290 
291291  User-agent: PerplexityBot
292+ Allow: /sp/
292293  Disallow: /
293- Allow: /sp/
294294 
295295  User-agent: PetalBot
296+ Allow: /sp/
296297  Disallow: /
297- Allow: /sp/
298298 
299299  User-agent: Primalbot
300+ Allow: /sp/
300301  Disallow: /
301- Allow: /sp/
302302 
303303  User-agent: python-requests
304+ Allow: /sp/
304305  Disallow: /
305- Allow: /sp/
306306 
307307  User-agent: Re-Animator
308+ Allow: /sp/
308309  Disallow: /
309- Allow: /sp/
310310 
311311  User-agent: Riddler
312+ Allow: /sp/
312313  Disallow: /
313- Allow: /sp/
314314 
315315  User-agent: SafeDNSBot
316+ Allow: /sp/
316317  Disallow: /
317- Allow: /sp/
318318 
319319  User-agent: SBIntuitionsBot
320+ Allow: /sp/
320321  Disallow: /
321- Allow: /sp/
322322 
323323  User-agent: Scrapy
324+ Allow: /sp/
324325  Disallow: /
325- Allow: /sp/
326326 
327327  User-agent: SearchAtlas
328+ Allow: /sp/
328329  Disallow: /
329- Allow: /sp/
330330 
331331  User-agent: SemanticScholarBot
332+ Allow: /sp/
332333  Disallow: /
333- Allow: /sp/
334334 
335335  User-agent: SemrushBot
336+ Allow: /sp/
336337  Disallow: /
337- Allow: /sp/
338338 
339339  User-agent: SentiBot
340+ Allow: /sp/
340341  Disallow: /
341- Allow: /sp/
342342 
343343  User-agent: Sogou
344+ Allow: /sp/
344345  Disallow: /
345- Allow: /sp/
346346 
347347  User-agent: Spider-Pig
348+ Allow: /sp/
348349  Disallow: /
349- Allow: /sp/
350350 
351351  User-agent: sqlmap
352+ Allow: /sp/
352353  Disallow: /
353- Allow: /sp/
354354 
355355  User-agent: StorygizeBot
356+ Allow: /sp/
356357  Disallow: /
357- Allow: /sp/
358358 
359359  User-agent: Test Certificate Info
360+ Allow: /sp/
360361  Disallow: /
361- Allow: /sp/
362362 
363363  User-agent: TikTokSpider
364+ Allow: /sp/
364365  Disallow: /
365- Allow: /sp/
366366 
367367  User-agent: Timpibot
368+ Allow: /sp/
368369  Disallow: /
369- Allow: /sp/
370370 
371371  User-agent: Turnitin
372+ Allow: /sp/
372373  Disallow: /
373- Allow: /sp/
374374 
375375  User-agent: uipbot
376+ Allow: /sp/
376377  Disallow: /
377- Allow: /sp/
378378 
379379  User-agent: vBSEO
380+ Allow: /sp/
380381  Disallow: /
381- Allow: /sp/
382382 
383383  User-agent: VegeBot
384+ Allow: /sp/
384385  Disallow: /
385- Allow: /sp/
386386 
387387  User-agent: VelenPublicWebCrawler
388+ Allow: /sp/
388389  Disallow: /
389- Allow: /sp/
390390 
391391  User-agent: Web Link Validator
392+ Allow: /sp/
392393  Disallow: /
393- Allow: /sp/
394394 
395395  User-agent: WebCruiser
396+ Allow: /sp/
396397  Disallow: /
397- Allow: /sp/
398398 
399399  User-agent: webzio-extended
400+ Allow: /sp/
400401  Disallow: /
401- Allow: /sp/
402402 
403403  User-agent: Xenu
404+ Allow: /sp/
404405  Disallow: /
405- Allow: /sp/
406406 
407407  User-agent: YouBot
408+ Allow: /sp/
408409  Disallow: /
409- Allow: /sp/
410410 
411411  User-agent: *
412412  Disallow: /wp-admin/