NeuralCrawl

Nikkei / robots.txt snapshot

← back to nikkei.com · fetched 2026-06-20T01:10:30Z (1mo ago) · HTTP 200 · 2149 bytes · sha256 ea59ffd52c9dec79 · raw

final URL: https://www.nikkei.com/robots.txt

1User-agent: *
2Allow: /markets/kabu/stkcomp/$
3Allow: /markets/kabu/gyoshucomp/$
4Disallow: /.tools
5Disallow: /markets/kabu/stkcomp/
6Disallow: /markets/kabu/gyoshucomp/
7Disallow: /.resources/k-components/rtoaster/
8
9# If you want to use our data in machine learning, please contact us from the following url.
10# ja: https://nkbb.nikkei.co.jp/article/
11# en: https://nkbb.nikkei.co.jp/en/dataset/nikkei-news-articles
12
13User-agent: CCBot
14Disallow: /
15
16User-agent: GPTBot
17Disallow: /
18
19User-agent: ChatGPT-User
20Disallow: /
21
22User-agent: Google-Extended
23Disallow: /
24
25User-agent: anthropic-ai
26Disallow: /
27
28User-agent: cohere-ai
29Disallow: /
30
31User-Agent: omgili
32Disallow: /
33
34User-Agent: omgilibot
35Disallow: /
36
37User-Agent: ICC-Crawler
38Disallow: /
39
40User-agent: Applebot-Extended
41Disallow: /
42
43User-agent: ClaudeBot
44Disallow: /
45
46User-agent: Claude-Web
47Disallow: /
48
49User-agent: Claude-SearchBot
50Disallow: /
51
52User-agent: Claude-User
53Disallow: /
54
55User-agent: PerplexityBot
56Disallow: /
57
58User-agent: Perplexity-ai
59Disallow: /
60
61User-agent: Perplexity-User
62Disallow: /
63
64User-agent: Bytespider
65Disallow: /
66
67User-agent: Diffbot
68Disallow: /
69
70User-agent: FacebookBot
71Disallow: /
72
73User-agent: OAI-SearchBot
74Disallow: /
75
76User-agent: MJ12bot
77Disallow: /
78
79User-agent: PiplBot
80Disallow: /
81
82User-agent: Meta-ExternalAgent
83Disallow: /
84
85User-agent: Timpibot
86Disallow: /
87
88User-agent: Webzio
89Disallow: /
90
91User-agent: Webzio-extended
92Disallow: /
93
94User-agent: AmazonBot
95Disallow: /
96
97User-agent: cohere-training-data-crawler
98Disallow: /
99
100User-agent: magpie-crawler
101Disallow: /
102
103User-agent: meta-externalfetcher
104Disallow: /
105
106User-agent: Scrapy
107Disallow: /
108
109User-agent: DuckAssistBot
110Disallow: /
111
112User-agent: Google-CloudVertexBot
113Disallow: /
114
115User-agent: AI2Bot
116Disallow: /
117
118User-agent: AI2Bot-Dolma
119Disallow: /
120
121User-agent: img2dataset
122Disallow: /
123
124User-agent: YouBot
125Disallow: /
126
127User-agent: SeekrBot
128Disallow: /
129
130User-agent: OpenindexSpider
131Disallow: /
132
133User-agent: Gensparkbot
134Disallow: /
135
136User-agent: SBIntuitionsBot
137Disallow: /
138
139User-agent: SBIntuitions-SearchBot
140Disallow: /
141
142Sitemap: https://www.nikkei.com/sitemap.xml
143Sitemap: https://www.nikkei.com/sitemaps/topics_sitemap_index.xml