- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2609snapshots
1437changes detected
226this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Allow: / for *
- Removed Disallow: /*?brand= for *
- Removed Disallow: /*?condition= for *
- Removed Disallow: /*?lifestage= for *
- Removed Disallow: /*?lightboxfired= for *
- Removed Disallow: /*?productform= for *
- … 4 more change(s)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- First snapshot of robots.txt archived
- First snapshot of robots.txt archived
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.snapchat.com/sitemaps/entertainment_funny/sitemap-index.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)