- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2634snapshots
1462changes detected
251this week
- Removed Disallow: /de-at/docs for *
- Removed Disallow: /de-ch/docs for *
- Removed Disallow: /de-de/docs for *
- Removed Disallow: /en-au/docs for *
- Removed Disallow: /en-be/docs for *
- Removed Disallow: /en-ca/docs for *
- … 39 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /*/*/preview-cms/* for *
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘Brightbot’ entirely
- 🤖 AI Blocked omgili (Webz.io) entirely
- Blocked user-agent ‘omgilibot’ entirely
- 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
- 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
- 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
- … 45 more change(s)
- 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
- 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
- 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Allow: /i/api/* for Bingbot
- Added Disallow: /*/likes for Bingbot
- Added Disallow: /*/likes? for Bingbot
- Added Disallow: /*/media for Bingbot
- Added Disallow: /*/media? for Bingbot
- Added Disallow: /*/photo$ for Bingbot
- … 40 more change(s)
- Added rules for user-agent ‘AdsBot-Google’
- Added rules for user-agent ‘AdsBot-Google-Mobile’
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- 🤖 AI Blocked Bytespider (ByteDance) entirely
- 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
- 🤖 AI Blocked Claude-Web (Anthropic) entirely
- 🤖 AI Blocked cohere-ai (Cohere) entirely
- … 45 more change(s)
- Added Disallow: /potash/co-op-program684217$ for *
- Removed Disallow: /potash/co-op-program$ for *
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)