NeuralCrawl

Akamai Technologies / robots.txt snapshot

← back to akamai.com · fetched 2026-06-20T01:10:31Z (18h ago) · HTTP 200 · 1670 bytes · sha256 a87805ba313c63a8 · raw

final URL: https://www.akamai.com/robots.txt

1# Last updated 20 April 2026
2
3Sitemap: https://www.akamai.com/sitemap-index.xml
4
5# Allow AI Agents to crawl everything
6User-agent: AliyunBot
7User-agent: AliyunCrawler
8User-agent: AliyunSecBot
9User-agent: AlibabaCloud
10User-agent: Amazonbot
11User-agent: anthropic-ai
12User-agent: Applebot
13User-agent: BaichuanBot
14User-agent: Baidu-YiYan
15User-agent: BaiduAI
16User-agent: BaiduApp
17User-agent: BaiduSmartProgram
18User-agent: Baiduspider
19User-agent: Bingbot
20User-agent: BingPreview
21User-agent: Bytespider
22User-agent: CCBot
23User-agent: ChatGPT-User
24User-agent: Claude-User
25User-agent: Claude-web
26User-agent: ClaudeBot
27User-agent: Claude-SearchBot
28User-agent: DeepSeek-Crawler
29User-agent: DeepSeekBot
30User-agent: DuckDuckBot
31User-agent: ERNIE-Bot
32User-agent: ERNIEBot
33User-agent: FacebookBot
34User-agent: Google-Extended
35User-agent: Google-NotebookLM
36User-agent: GoogleOther
37User-agent: GoogleOther-Image
38User-agent: GoogleOther-Video
39User-agent: GPTBot
40User-agent: ia-archiver
41User-agent: iFlytekBot
42User-agent: KimiBot
43User-agent: MiniMaxBot
44User-agent: MoonshotAI
45User-agent: MoonshotBot
46User-agent: OAI-SearchBot
47User-agent: Perplexity-User
48User-agent: PerplexityBot
49User-agent: Qwen
50User-agent: Sogou
51User-agent: Sogou spider
52User-agent: TencentTraveler
53User-agent: TencentBot
54User-agent: Twitterbot
55User-agent: ZhipuBot
56Allow: /
57
58# Keep specific restrictions for everyone else
59User-agent: *
60Disallow: /x-failover/
61Disallow: /fr/company/locations/
62Disallow: /it/company/locations/
63Disallow: /es/company/locations/
64Disallow: /pt/company/locations/
65Disallow: /ja/company/locations/
66Disallow: /zh/company/locations/
67Disallow: /ko/company/locations/
68Disallow: */resources/content/*