NeuralCrawl

AstraZeneca / robots.txt snapshot

← back to astrazeneca.com · fetched 2026-08-17T17:05:52Z (1mo ago) · HTTP 200 · 910 bytes · sha256 93019084380ecfce · raw

final URL: https://www.astrazeneca.com/robots.txt

1User-agent: Googlebot
2Allow: /
3
4User-agent: Bingbot
5Allow: /
6
7User-agent: GPTBot
8Allow: /
9
10User-agent: OAI-SearchBot
11Allow: /
12
13User-agent: ChatGPT-User
14Allow: /
15
16User-agent: ClaudeBot
17Allow: /
18
19User-agent: Claude-SearchBot
20Allow: /
21
22User-agent: Claude-User
23Allow: /
24
25User-agent: Google-Extended
26Allow: /
27
28User-agent: PerplexityBot
29Allow: /
30
31User-agent: Perplexity-User
32Allow: /
33
34User-agent: Applebot
35Allow: /
36
37User-agent: Applebot-Extended
38Allow: /
39
40User-agent: DuckAssistBot
41Allow: /
42
43User-agent: MistralAI-User
44Allow: /
45
46User-agent: Amazonbot
47Allow: /
48
49User-agent: Meta-ExternalAgent
50Allow: /
51
52User-agent: CCBot
53Allow: /
54
55User-agent: YouBot
56Allow: /
57
58User-agent: PhindBot
59Allow: /
60
61User-agent: AndiBot
62Allow: /
63
64User-agent: FirecrawlAgent
65Allow: /
66
67User-agent: Exabot
68Allow: /
69
70User-agent: *
71Disallow: /*content/astraz*
72Disallow: /*_escaped_fragment_
73
74Sitemap: https://www.astrazeneca.com/azcomsitemap.xml