The Pulse

Every change to a monitored robots.txt, archived and explained.

1166sites tracked

1052files archived

1071snapshots

19changes detected

19this week

All changes 🤖 AI-crawler changes Hiding initial archives

Advance Auto Parts advanceautoparts.com modified +0 −27 5h ago

Removed all rules for user-agent ‘*’
Removed all rules for user-agent ‘Mediapartners-Google’
Removed sitemap https://shop.advanceautoparts.com/sitemap.xml

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 5h ago

Added sitemap http://127.0.0.1/image-sitemap.xml
Added sitemap http://127.0.0.1/page-sitemap.xml
Removed sitemap http://www.iqvia.com/image-sitemap.xml
Removed sitemap http://www.iqvia.com/page-sitemap.xml

view diff → snapshot

F5 f5.com modified +1 −1 5h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Glencore glencore.com modified +5 −5 5h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Adidas adidas.com modified +1 −1 5h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Goodyear Tire & Rubber goodyear.com modified +40 −0 5h ago

Blocked user-agent ‘ahrefsbot’ entirely
Blocked user-agent ‘cazoodlebot’ entirely
Blocked user-agent ‘dotbot/1.0’ entirely
Blocked user-agent ‘gigabot’ entirely
Blocked user-agent ‘mj12bot’ entirely
Blocked user-agent ‘sogou inst spider’ entirely
… 20 more change(s)

view diff → snapshot

Amgen amgen.com modified +2 −0 5h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 5h ago

Added sitemap https://www.bca.co.id/sitemap.xml
Removed sitemap http://www.bca.co.id/sitemap.xml

view diff → snapshot

Toronto Star thestar.com modified +15 −15 5h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Glencore glencore.com modified +5 −5 5h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Goodyear Tire & Rubber goodyear.com modified +0 −40 8h ago

Removed all rules for user-agent ‘ahrefsbot’
Removed all rules for user-agent ‘cazoodlebot’
Removed all rules for user-agent ‘dotbot/1.0’
Removed all rules for user-agent ‘gigabot’
Removed all rules for user-agent ‘mj12bot’
Removed all rules for user-agent ‘sogou inst spider’
… 20 more change(s)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 8h ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

Amgen amgen.com modified +0 −2 8h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +15 −15 8h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 11h ago

Added sitemap http://www.bca.co.id/sitemap.xml
Removed sitemap https://www.bca.co.id/sitemap.xml

view diff → snapshot

Toronto Star thestar.com modified +15 −15 11h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Glencore glencore.com modified +5 −5 11h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +16 −16 17h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Glencore glencore.com modified +5 −5 17h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot