NeuralCrawl

The Asahi Shimbun / robots.txt snapshot

← back to asahi.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 2021 bytes · sha256 705a62a8c7f7a3f9 · raw

final URL: https://www.asahi.com/robots.txt

1User-Agent: *
2Disallow: /travel/event/search/
3Disallow: /car/index.html
4Disallow: /housing/index.html
5Disallow: /english/newsfeatures.html
6Disallow: /english/business.html
7Disallow: /english/cooljapan.html
8Disallow: /english/sports.html
9Disallow: /*/search/results*
10Allow: /
11Allow: /.well-known/assetlinks.json
12Allow: /ads/
13
14
15User-agent: Googlebot
16Disallow: /*klpuid=*
17Allow: /ads/
18
19User-agent: CCBot
20Disallow: /
21
22User-agent: OAI-SearchBot
23Disallow: /
24
25User-agent: ChatGPT-User
26Disallow: /
27Allow: /ads/
28
29User-agent: GPTBot
30Disallow: /
31Allow: /ads/
32
33User-agent: Google-Extended
34Disallow: /
35Allow: /ads/
36
37User-agent: Google-CloudVertexBot
38Disallow: /
39Allow: /ads/
40
41User-agent: ICC-Crawler
42Disallow: /
43Allow: /ads/
44
45User-agent: anthropic-ai
46Disallow: /
47Allow: /ads/
48
49User-agent: ClaudeBot
50Disallow: /
51Allow: /ads/
52
53User-agent: Claude-Web
54Disallow: /
55Allow: /ads/
56
57User-agent: Claude-SearchBot
58Disallow: /
59Allow: /ads/
60
61User-agent: Claude-User
62Disallow: /
63Allow: /ads/
64
65User-agent: Applebot-Extended
66Disallow: /
67Allow: /ads/
68
69User-agent: cohere-ai
70Disallow: /
71Allow: /ads/
72
73User-agent: Cohere-training-data-crawler
74Disallow: /
75Allow: /ads/
76
77User-agent: omgili
78Disallow: /
79Allow: /ads/
80
81User-agent: omgilibot
82Disallow: /
83Allow: /ads/
84
85User-agent: PerplexityBot
86Disallow: /
87
88User-agent: Perplexity-ai
89Disallow: /
90
91User-agent: Perplexity-User
92Disallow: /
93
94User-agent: FacebookBot
95Disallow: /
96Allow: /ads/
97
98User-agent: Meta-ExternalAgent
99Disallow: /
100Allow: /ads/
101
102User-agent: Meta-externalfetcher
103Disallow: /
104Allow: /ads/
105
106User-agent: Bytespider
107Disallow: /
108
109User-agent: Gensparkbot
110Disallow: /
111
112User-agent: AmazonBot
113Disallow: /
114Allow: /ads/
115
116User-agent: Diffbot
117Disallow: /
118
119User-agent: Magpie-crawler
120Disallow: /
121
122User-agent: Scrapy
123Disallow: /
124
125User-agent: Timpibot
126Disallow: /
127
128User-agent: Webzio-Extended
129Disallow: /
130
131User-agent: SBIntuitionsBot
132Disallow: /
133
134User-agent: SBIntuitions-SearchBot
135Disallow: /
136
137User-agent: OpenindexSpider
138Disallow: /
139
140User-agent: AI2Bot
141Disallow: /
142
143sitemap: https://www.asahi.com/sitemap.xml