Nikkei / robots.txt
← back to nikkei.com · change detected 9d ago (2026-06-17T02:02:49Z) · modified +143 −10
What changed
- Response is a valid robots.txt again (was an HTML page)
- Added rules for user-agent ‘*’
- 🤖 AIBlocked AI2Bot (Allen Institute for AI) entirely
- Blocked user-agent ‘AI2Bot-Dolma’ entirely
- 🤖 AIBlocked Amazonbot (Amazon) entirely
- 🤖 AIBlocked anthropic-ai (Anthropic) entirely
- 🤖 AIBlocked Applebot-Extended (Apple) entirely
- 🤖 AIBlocked Bytespider (ByteDance) entirely
- 🤖 AIBlocked CCBot (Common Crawl) entirely
- 🤖 AIBlocked ChatGPT-User (OpenAI) entirely
- 🤖 AIBlocked Claude-SearchBot (Anthropic) entirely
- 🤖 AIBlocked Claude-User (Anthropic) entirely
- 🤖 AIBlocked Claude-Web (Anthropic) entirely
- 🤖 AIBlocked ClaudeBot (Anthropic) entirely
- 🤖 AIBlocked cohere-ai (Cohere) entirely
- Blocked user-agent ‘cohere-training-data-crawler’ entirely
- 🤖 AIBlocked Diffbot (Diffbot) entirely
- 🤖 AIBlocked DuckAssistBot (DuckDuckGo) entirely
- 🤖 AIBlocked FacebookBot (Meta) entirely
- Blocked user-agent ‘Gensparkbot’ entirely
- Blocked user-agent ‘Google-CloudVertexBot’ entirely
- 🤖 AIBlocked Google-Extended (Google) entirely
- 🤖 AIBlocked GPTBot (OpenAI) entirely
- Blocked user-agent ‘ICC-Crawler’ entirely
- Blocked user-agent ‘img2dataset’ entirely
- Blocked user-agent ‘magpie-crawler’ entirely
- 🤖 AIBlocked meta-externalagent (Meta) entirely
- 🤖 AIBlocked meta-externalfetcher (Meta) entirely
- Blocked user-agent ‘MJ12bot’ entirely
- 🤖 AIBlocked OAI-SearchBot (OpenAI) entirely
- 🤖 AIBlocked omgili (Webz.io) entirely
- Blocked user-agent ‘omgilibot’ entirely
- Blocked user-agent ‘OpenindexSpider’ entirely
- Blocked user-agent ‘Perplexity-ai’ entirely
- 🤖 AIBlocked Perplexity-User (Perplexity) entirely
- 🤖 AIBlocked PerplexityBot (Perplexity) entirely
- Blocked user-agent ‘PiplBot’ entirely
- Blocked user-agent ‘SBIntuitions-SearchBot’ entirely
- Blocked user-agent ‘SBIntuitionsBot’ entirely
- Blocked user-agent ‘Scrapy’ entirely
- Blocked user-agent ‘SeekrBot’ entirely
- 🤖 AIBlocked Timpibot (Timpi) entirely
- Blocked user-agent ‘Webzio’ entirely
- Blocked user-agent ‘Webzio-extended’ entirely
- 🤖 AIBlocked YouBot (You.com) entirely
- Added sitemap https://www.nikkei.com/sitemap.xml
- Added sitemap https://www.nikkei.com/sitemaps/topics_sitemap_index.xml
Diff (no previous version) → (file removed)
No line-level differences (identical or both empty).