- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2647snapshots
1476changes detected
231this week
- Formatting/comment-only change (no semantic difference)
- Removed sitemap https://corporate.arcelormittal.com/sitemap.xml
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘Brightbot’ entirely
- 🤖 AI Blocked omgili (Webz.io) entirely
- Blocked user-agent ‘omgilibot’ entirely
- 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
- 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
- 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
- … 45 more change(s)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Blocked Bytespider (ByteDance) entirely
- Blocked user-agent ‘Diffbot-User’ entirely
- Blocked user-agent ‘Scrapy’ entirely
- 🤖 AI Blocked Timpibot (Timpi) entirely
- 🤖 AI Blocked YouBot (You.com) entirely
- Added Disallow: /marketing/prerelease/ for *
- Added Disallow: /marketing/prerelease/ for Googlebot
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- 🤖 AI Blocked Bytespider (ByteDance) entirely
- 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
- 🤖 AI Blocked Claude-Web (Anthropic) entirely
- 🤖 AI Blocked cohere-ai (Cohere) entirely
- … 45 more change(s)
- Added sitemap https://www.rgare.com/docs/default-source/sitemaps/sitemap-index.xml
- Removed sitemap https://www.rgare.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for *
- Removed Disallow: /H1Rj9-8bvN3I9mQlIPVT5Cm31oOG5HamhkmPLNbEbyE=.html for *
- Added Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for DotBot
- Removed Disallow: /H1Rj9-8bvN3I9mQlIPVT5Cm31oOG5HamhkmPLNbEbyE=.html for DotBot
- Added Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for sistrix
- Removed Disallow: /H1Rj9-8bvN3I9mQlIPVT5Cm31oOG5HamhkmPLNbEbyE=.html for sistrix
- … 2 more change(s)
- 🤖 AI Added Allow: /rss-feeds/feeds/bf-amazon-ai-grounding.extended.xml for Amazonbot
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /Global-reach/Europe/United-Kingdom/Legal-regulatory-hub/Almus-NetGen-Terms-Conditions for *
- Added Disallow: /Global-reach/Europe/United-Kingdom/Legal-regulatory-hub/BE-Force-terms-conditions for *
- Added Disallow: /Global-reach/Europe/United-Kingdom/Legal-regulatory-hub/pharmacyone-brand-equalisation-terms for *
- Removed Disallow: /web-team-requests for *
- Added Disallow: /gen/ for *
- Added Disallow: /gen/ for Googlebot
- Added Disallow: /studio-events for *
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Added sitemap https://www.lilly.com/uk/sitemap-index.xml
- Formatting/comment-only change (no semantic difference)