NeuralCrawl

Naspers / robots.txt snapshot

← back to naspers.com · fetched 2026-06-20T01:10:30Z (15h ago) · HTTP 200 · 1099 bytes · sha256 1b9c3ec68a448b1a · raw

final URL: https://www.naspers.com/robots.txt

1User-agent: AhrefsBot
2Disallow: /
3User-agent: Insieve+Bot
4Disallow: /
5User-agent: gsa-crawler+(Enterprise;+T3-HR8MK3S756ETJ;[email protected])
6Disallow: /
7User-agent: BLP_bbot/0.1
8Disallow: /
9User-agent: blp_bbot/0.1
10Disallow: /
11User-agent: Java/1.6.0
12Disallow: /
13User-agent: Java/1.6.0_04
14Disallow: /
15User-agent: Java/1.6.0_11
16Disallow: /
17User-agent: Java/1.6.0_15
18Disallow: /
19User-agent: Java/1.6.0_16
20Disallow: /
21User-agent: Java/1.6.0_17
22Disallow: /
23User-agent: Java/1.6.0_18
24Disallow: /
25User-agent: Java/1.6.0_20
26Disallow: /
27User-agent: Java/1.6.0_22
28Disallow: /
29User-agent: Java/1.6.0_24
30Disallow: /
31User-agent: Java/1.6.0_25
32Disallow: /
33User-agent: *
34Disallow: /staticresources/
35Disallow: /investis/
36Disallow: /tools/
37Disallow: /investis modules/
38Disallow: /*/sitecore/
39Disallow: /sitecore/
40Disallow: /*/sitecore/content/
41Disallow: /sitecore/content/
42Disallow: /error-page.aspx
43Disallow: /*/error-page.aspx
44Disallow: /favicon.ico
45Disallow: /WebResource.axd
46Disallow: /Webresource.axd
47Disallow: /ScriptResource.axd
48
49#XML sitemap
50sitemap: https://www.naspers.com/sitemap.xml