NeuralCrawl

Nikkei / robots.txt snapshot

← back to nikkei.com · fetched 2026-10-29T05:14:04Z (9d ago) · HTTP 200 · 2617 bytes · sha256 5d64ad9b397e034b · raw

final URL: https://www.nikkei.com/robots.txt

1User-agent: *
2Allow: /markets/kabu/stkcomp/$
3Allow: /markets/kabu/gyoshucomp/$
4Disallow: /.tools
5Disallow: /markets/kabu/stkcomp/
6Disallow: /markets/kabu/gyoshucomp/
7Disallow: /.resources/k-components/rtoaster/
8
9# If you want to use our data in machine learning, please contact us from the following url.
10# ja: https://nkbb.nikkei.co.jp/article/
11# en: https://nkbb.nikkei.co.jp/en/dataset/nikkei-news-articles
12
13User-agent: GPTBot
14User-agent: OAI-SearchBot
15User-agent: ChatGPT-User
16User-agent: ClaudeBot
17User-agent: Claude-SearchBot
18User-agent: Claude-User
19User-agent: Google-Extended
20Disallow: /
21Allow: /promotion/$
22Allow: /promotion/family/$
23Allow: /promotion/onboarding/guide_r3/$
24Allow: /promotion/onboarding/guide_r3/story_shukatsu_01/$
25Allow: /promotion/onboarding/guide_r3/method_news/$
26Allow: /promotion/campaign/student/startguide/$
27Allow: /promotion/about/$
28Allow: /promotion/houjin/$
29Allow: /promotion/houjin/price/$
30Allow: /promotion/houjin/welfare/$
31Allow: /promotion/houjin/location/$
32Allow: /promotion/houjin/house-organ/$
33Allow: /promotion/houjin/casestudies/
34Allow: /promotion/houjin/download/
35Allow: /promotion/houjin/guide/
36
37User-agent: CCBot
38Disallow: /
39
40User-agent: anthropic-ai
41Disallow: /
42
43User-agent: cohere-ai
44Disallow: /
45
46User-Agent: omgili
47Disallow: /
48
49User-Agent: omgilibot
50Disallow: /
51
52User-Agent: ICC-Crawler
53Disallow: /
54
55User-agent: Applebot-Extended
56Disallow: /
57
58User-agent: Claude-Web
59Disallow: /
60
61User-agent: PerplexityBot
62Disallow: /
63
64User-agent: Perplexity-ai
65Disallow: /
66
67User-agent: Perplexity-User
68Disallow: /
69
70User-agent: Bytespider
71Disallow: /
72
73User-agent: Diffbot
74Disallow: /
75
76User-agent: FacebookBot
77Disallow: /
78
79User-agent: MJ12bot
80Disallow: /
81
82User-agent: PiplBot
83Disallow: /
84
85User-agent: Meta-ExternalAgent
86Disallow: /
87
88User-agent: Timpibot
89Disallow: /
90
91User-agent: Webzio
92Disallow: /
93
94User-agent: Webzio-extended
95Disallow: /
96
97User-agent: AmazonBot
98Disallow: /
99
100User-agent: cohere-training-data-crawler
101Disallow: /
102
103User-agent: magpie-crawler
104Disallow: /
105
106User-agent: meta-externalfetcher
107Disallow: /
108
109User-agent: Scrapy
110Disallow: /
111
112User-agent: DuckAssistBot
113Disallow: /
114
115User-agent: Google-CloudVertexBot
116Disallow: /
117
118User-agent: AI2Bot
119Disallow: /
120
121User-agent: AI2Bot-Dolma
122Disallow: /
123
124User-agent: img2dataset
125Disallow: /
126
127User-agent: YouBot
128Disallow: /
129
130User-agent: SeekrBot
131Disallow: /
132
133User-agent: OpenindexSpider
134Disallow: /
135
136User-agent: Gensparkbot
137Disallow: /
138
139User-agent: SBIntuitionsBot
140Disallow: /
141
142User-agent: SBIntuitions-SearchBot
143Disallow: /
144
145Sitemap: https://www.nikkei.com/sitemap.xml
146Sitemap: https://www.nikkei.com/sitemaps/topics_sitemap_index.xml