- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2673snapshots
1503changes detected
231this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- Blocked user-agent ‘Applebot’ entirely
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- Blocked user-agent ‘Baiduspider’ entirely
- Blocked user-agent ‘Baiduspider-image’ entirely
- Blocked user-agent ‘Baiduspider-video’ entirely
- … 13 more change(s)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Added sitemap https://www.adobe.com/edu/sitemap-index.xml
- Removed sitemap https://www.adobe.com/edu-shared/assets/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.thetradedesk.com/sitemap.xml
- Removed sitemap https://careers.thetradedesk.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- robots.txt is no longer served (HTTP 404)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Added sitemap https://www.lilly.com/medical/sitemap-index.xml
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /*/media/oembed for *
- Added Disallow: /README.md for *
- Added Disallow: /composer/Metapackage/README.txt for *
- Added Disallow: /composer/Plugin/ProjectMessage/README.md for *
- Added Disallow: /composer/Plugin/Scaffold/README.md for *
- Added Disallow: /composer/Plugin/VendorHardening/README.txt for *
- … 25 more change(s)
- robots.txt is no longer served (HTTP 404)
- First snapshot of robots.txt archived
- First snapshot of robots.txt archived
- First snapshot of robots.txt archived
- First snapshot of robots.txt archived
- First snapshot of robots.txt archived