- Added sitemap https://totalenergies.com/sitemap.xml
- Removed sitemap https://prd-backoffice.totalenergies.com/sitemap.xml
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5428snapshots
4292changes detected
227this week
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: */search* for *
- Added rules for user-agent ‘OAI-AdsBot’
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Removed Disallow: /favex? for *
- Removed Disallow: /loginx for *
- Removed Disallow: /replyx? for *
- Removed Disallow: /votex? for *
- Added sitemap https://www.autonation.com/sitemap/srp/condition_make_location
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- Added Disallow: /sites/default/files/white-papers/pfpt-au-wp-securing-ai-agents-guide.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-br-wp-securing-ai-agents-guide.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-de-wp-securing-ai-agents-guide.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-es-wp-securing-ai-agents-guide.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-fr-wp-securing-ai-agents-guide.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-it-wp-securing-ai-agents-guide.pdf for *
- … 2 more change(s)
- robots.txt is no longer served (HTTP 404)
- Added Disallow: /api/article-search for *
- Added Disallow: /xid for *
- Removed sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)