The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1206files archived

5433snapshots

4298changes detected

233this week

All changes 🤖 AI-crawler changes Hiding initial archives

Toronto Star thestar.com modified +17 −17 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

DSV dsv.com modified +25 −25 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Amgen amgen.com modified +0 −2 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 3mo ago

Added sitemap https://www.bca.co.id/sitemap.xml
Removed sitemap http://www.bca.co.id/sitemap.xml

view diff → snapshot

WordPress wordpress.com modified +1 −1 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +13 −13 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 3mo ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

KLA kla.com modified +1 −0 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +14 −14 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Advance Auto Parts advanceautoparts.com modified +27 −0 3mo ago

Added rules for user-agent ‘*’
Added rules for user-agent ‘Mediapartners-Google’
Added sitemap https://shop.advanceautoparts.com/sitemap.xml

view diff → snapshot

DSV dsv.com modified +28 −28 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 3mo ago

Added sitemap http://www.bca.co.id/sitemap.xml
Removed sitemap https://www.bca.co.id/sitemap.xml

view diff → snapshot

Toronto Star thestar.com modified +15 −15 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Pinterest pinterest.com modified +5 −5 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Facebook facebook.com modified +329 −21 3mo ago

🤖 AI Unblocked Amazonbot (Amazon)
🤖 AI Added Allow: /ajax/bootloader-endpoint/ for Amazonbot
🤖 AI Added Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet for Amazonbot
🤖 AI Added Allow: /careers/ for Amazonbot
🤖 AI Added Allow: /safetycheck/ for Amazonbot
🤖 AI Added Disallow: /*/plugins/* for Amazonbot
… 45 more change(s)

view diff → snapshot

Adidas adidas.com modified +1 −1 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Sonic Automotive sonicautomotive.com modified +0 −3 3mo ago

Removed all rules for user-agent ‘Google-InspectionTool’

view diff → snapshot

Advance Auto Parts advanceautoparts.com modified +0 −27 3mo ago

Removed all rules for user-agent ‘*’
Removed all rules for user-agent ‘Mediapartners-Google’
Removed sitemap https://shop.advanceautoparts.com/sitemap.xml

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 3mo ago

Added sitemap https://www.bca.co.id/sitemap.xml
Removed sitemap http://www.bca.co.id/sitemap.xml

view diff → snapshot

WordPress wordpress.com modified +1 −1 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +15 −15 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Meta Platforms meta.com modified +25 −0 3mo ago

Added sitemap https://www.meta.com/sitemap/www_meta_com_sitemap_da_dk.xml.gz
Added sitemap https://www.meta.com/sitemap/www_meta_com_sitemap_de_at.xml.gz
Added sitemap https://www.meta.com/sitemap/www_meta_com_sitemap_de_ch.xml.gz
Added sitemap https://www.meta.com/sitemap/www_meta_com_sitemap_de_de.xml.gz
Added sitemap https://www.meta.com/sitemap/www_meta_com_sitemap_en_au.xml.gz
Added sitemap https://www.meta.com/sitemap/www_meta_com_sitemap_en_ca.xml.gz
… 19 more change(s)

view diff → snapshot

DSV dsv.com modified +25 −25 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +17 −17 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 3mo ago

Added sitemap http://127.0.0.1/image-sitemap.xml
Added sitemap http://127.0.0.1/page-sitemap.xml
Removed sitemap http://www.iqvia.com/image-sitemap.xml
Removed sitemap http://www.iqvia.com/page-sitemap.xml

view diff → snapshot

KLA kla.com modified +1 −0 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

DSV dsv.com modified +22 −22 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Amgen amgen.com modified +2 −0 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot