- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
- Added Disallow: /marketplace/az/* for *
- Added Disallow: /marketplace/ca/* for *
- Added Disallow: /marketplace/fl/* for *
- Added Disallow: /marketplace/ga/* for *
- Added Disallow: /marketplace/ia/* for *
- … 23 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘*’
- Removed sitemap https://www.copart-sitemaps.com/ar/sitemap-index.xml
- Removed sitemap https://www.copart-sitemaps.com/es/sitemap-index.xml
- Removed sitemap https://www.copart-sitemaps.com/fr-CA/sitemap-index.xml
- Removed sitemap https://www.copart-sitemaps.com/pl/sitemap-index.xml
- Removed sitemap https://www.copart-sitemaps.com/ru/sitemap-index.xml
- … 1 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /api/article-search for *
- Added Disallow: /xid for *
- Removed sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Added Disallow: (empty) for *
- Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
- Removed Disallow: /marketplace/az/* for *
- Removed Disallow: /marketplace/ca/* for *
- Removed Disallow: /marketplace/fl/* for *
- Removed Disallow: /marketplace/ga/* for *
- … 23 more change(s)
- robots.txt is no longer served (HTTP 404)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://totalenergies.com/sitemap.xml
- Removed sitemap https://prd-backoffice.totalenergies.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /marketplace/az/* for *
- Added Disallow: /marketplace/ca/* for *
- Added Disallow: /marketplace/fl/* for *
- Added Disallow: /marketplace/ga/* for *
- Added Disallow: /marketplace/ia/* for *
- Added Disallow: /marketplace/id/* for *
- … 16 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)