- Removed Disallow: /4624/TheAtlanticOnline/* for *
- Removed Disallow: /4624/TheAtlanticOnline/* for AmazonAdBot
- Removed Disallow: /4624/TheAtlanticOnline/* for Applebot
- Removed Disallow: /4624/TheAtlanticOnline/* for ArcSpanScraper
- Removed Disallow: /4624/TheAtlanticOnline/* for BingBot
- 🤖 AI Removed Disallow: /4624/TheAtlanticOnline/* for ChatGPT-User
- … 8 more change(s)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5416snapshots
4280changes detected
237this week
- Formatting/comment-only change (no semantic difference)
- Removed Allow: /wp-admin/admin-ajax.php for *
- Removed Disallow: /wp-admin/ for *
- Added sitemap https://x.com/sitemaps/finance0
- Blocked user-agent ‘__usage_ai-training’ entirely
- Blocked user-agent ‘__usage_ai-training’ entirely
- Blocked user-agent ‘__usage_ai-training’ entirely
- Blocked user-agent ‘__usage_ai-training’ entirely
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- 🤖 AI Blocked Bytespider (ByteDance) entirely
- 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
- 🤖 AI Blocked Claude-SearchBot (Anthropic) entirely
- 🤖 AI Blocked Claude-User (Anthropic) entirely
- … 4 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /*/?_rsc= for *
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘AgentDataBot’ entirely
- Blocked user-agent ‘AGI Agent’ entirely
- Blocked user-agent ‘AIWebIndex-Agent’ entirely
- Blocked user-agent ‘AndiAgent’ entirely
- Blocked user-agent ‘artsdata-crawler’ entirely
- Blocked user-agent ‘BeansLLM-CorpusBot’ entirely
- … 42 more change(s)
- Added sitemap https://www.eset.com/hk-en/sitemap.xml
- Added sitemap https://www.eset.com/ua/news.xml
- Added sitemap https://www.eset.com/ua/sitemap.xml
- Added sitemap https://www.eset.com/vn/sitemap.xml
- Removed Disallow: /web.config for *
- Blocked user-agent ‘__usage_ai-training’ entirely
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- 🤖 AI Blocked anthropic-ai (Anthropic) entirely
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- Blocked user-agent ‘atlassian-bot’ entirely
- Blocked user-agent ‘AwarioRssBot’ entirely
- Blocked user-agent ‘AwarioSmartBot’ entirely
- … 32 more change(s)
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- Blocked user-agent ‘Amzn-SearchBot’ entirely
- Added Disallow: */all/* for *
- Added Disallow: */filters for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
- Added rules for user-agent ‘Brightbot’
- 🤖 AI Added rules for AI crawler Diffbot (Diffbot)
- 🤖 AI Added rules for AI crawler DuckAssistBot (DuckDuckGo)
- Added rules for user-agent ‘Gemini-AI’
- Added rules for user-agent ‘GoogleAgent-Mariner’
- … 36 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘ExaSearchBot’ entirely
- Blocked user-agent ‘KeenableBot’ entirely
- 🤖 AI Added Allow: /ja-jp/partners/find-a-service-partner/sun-asterisk/$ for CHATGPT-User
- 🤖 AI Added Allow: /partners/find-a-service-partner/sun-asterisk/$ for CHATGPT-User
- 🤖 AI Added Allow: /ja-jp/partners/find-a-service-partner/sun-asterisk/$ for Claude-SearchBot
- 🤖 AI Added Allow: /partners/find-a-service-partner/sun-asterisk/$ for Claude-SearchBot
- 🤖 AI Added Allow: /ja-jp/partners/find-a-service-partner/sun-asterisk/$ for Claude-User
- 🤖 AI Added Allow: /partners/find-a-service-partner/sun-asterisk/$ for Claude-User
- … 4 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)