- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2647snapshots
1476changes detected
231this week
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘Brightbot’ entirely
- 🤖 AI Blocked omgili (Webz.io) entirely
- Blocked user-agent ‘omgilibot’ entirely
- 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
- 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
- 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
- … 45 more change(s)
- 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
- 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
- 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Allow: /i/api/* for Bingbot
- Added Disallow: /*/likes for Bingbot
- Added Disallow: /*/likes? for Bingbot
- Added Disallow: /*/media for Bingbot
- Added Disallow: /*/media? for Bingbot
- Added Disallow: /*/photo$ for Bingbot
- … 40 more change(s)
- Added rules for user-agent ‘AdsBot-Google’
- Added rules for user-agent ‘AdsBot-Google-Mobile’
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- 🤖 AI Blocked Bytespider (ByteDance) entirely
- 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
- 🤖 AI Blocked Claude-Web (Anthropic) entirely
- 🤖 AI Blocked cohere-ai (Cohere) entirely
- … 45 more change(s)
- Added Disallow: /potash/co-op-program684217$ for *
- Removed Disallow: /potash/co-op-program$ for *
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- First snapshot of robots.txt archived
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘BomboraBot’
- Added Disallow: /*?id=*&type=mdc_* for Citoid
- Added Disallow: /*?type=mdc_*&id=* for Citoid
- Added Disallow: /Date(* for Citoid
- Added Disallow: /acct/* for Citoid
- Added Disallow: /article/AP* for Citoid
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)