NeuralCrawl

AstraZeneca / robots.txt

← back to astrazeneca.com · change detected 1mo ago (2026-08-17T17:05:52Z) · modified +68 −5

What changed

Diff old (2026-06-20T01:10:30Z) → new (2026-08-17T17:05:52Z)

@@ -1,10 +1,73 @@
1- User-agent: LinkedInBot
1+ User-agent: Googlebot
2+ Allow: /
3+
4+ User-agent: Bingbot
5+ Allow: /
6+
7+ User-agent: GPTBot
8+ Allow: /
9+
10+ User-agent: OAI-SearchBot
11+ Allow: /
12+
13+ User-agent: ChatGPT-User
14+ Allow: /
15+
16+ User-agent: ClaudeBot
17+ Allow: /
18+
19+ User-agent: Claude-SearchBot
20+ Allow: /
21+
22+ User-agent: Claude-User
23+ Allow: /
24+
25+ User-agent: Google-Extended
26+ Allow: /
27+
28+ User-agent: PerplexityBot
29+ Allow: /
30+
31+ User-agent: Perplexity-User
32+ Allow: /
33+
34+ User-agent: Applebot
35+ Allow: /
36+
37+ User-agent: Applebot-Extended
38+ Allow: /
39+
40+ User-agent: DuckAssistBot
41+ Allow: /
42+
43+ User-agent: MistralAI-User
44+ Allow: /
45+
46+ User-agent: Amazonbot
47+ Allow: /
48+
49+ User-agent: Meta-ExternalAgent
50+ Allow: /
51+
52+ User-agent: CCBot
53+ Allow: /
54+
55+ User-agent: YouBot
56+ Allow: /
57+
58+ User-agent: PhindBot
59+ Allow: /
60+
61+ User-agent: AndiBot
62+ Allow: /
63+
64+ User-agent: FirecrawlAgent
65+ Allow: /
66+
67+ User-agent: Exabot
268  Allow: /
369 
4- User-agent: Googlebot
5- Allow: /
6-
7- User-Agent: *
70+ User-agent: *
871  Disallow: /*content/astraz*
972  Disallow: /*_escaped_fragment_
1073