The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1206files archived

5439snapshots

4304changes detected

239this week

All changes 🤖 AI-crawler changes Hiding initial archives

Toronto Star thestar.com modified +15 −15 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

South China Morning Post scmp.com modified +2 −0 4mo ago

Added Disallow: /search/* for *
Added Disallow: /search?q=* for *

view diff → snapshot

University of Cape Town uct.ac.za modified +2 −2 4mo ago

Unblocked user-agent ‘Bingbot’
Added Disallow: (empty) for Bingbot
🤖 AI Unblocked OAI-SearchBot (OpenAI)
🤖 AI Added Disallow: (empty) for OAI-SearchBot

view diff → snapshot

Advance Auto Parts advanceautoparts.com modified +0 −27 4mo ago

Removed all rules for user-agent ‘*’
Removed all rules for user-agent ‘Mediapartners-Google’
Removed sitemap https://shop.advanceautoparts.com/sitemap.xml

view diff → snapshot

KLA kla.com modified +1 −0 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Honeywell International honeywell.com modified +27 −62 4mo ago

Added Disallow: /bin/edam/getfileservlet for *
Added Disallow: /pif/ for *
Added Disallow: /us/en/legal/cookie-notice?ref=* for *
Removed Disallow: /br/pt?utm* for *
Removed Disallow: /content/dam/honeywellbt/en/documents/downloads/hon-HoneywellForge_ForBuildings_Brochure_042020.pdf for *
Removed Disallow: /content/dam/honeywellbt/en/documents/downloads/honeywell-forge/Honeywell_Forge_Terms.pdf for *
… 45 more change(s)

view diff → snapshot

WordPress wordpress.com modified +1 −1 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +15 −15 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Corriere della Sera corriere.it modified +10 −0 4mo ago

🤖 AI Blocked Bytespider (ByteDance) entirely
Blocked user-agent ‘Diffbot-User’ entirely
Blocked user-agent ‘Scrapy’ entirely
🤖 AI Blocked Timpibot (Timpi) entirely
🤖 AI Blocked YouBot (You.com) entirely

view diff → snapshot

Amgen amgen.com modified +2 −0 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 4mo ago

Added sitemap http://www.bca.co.id/sitemap.xml
Removed sitemap https://www.bca.co.id/sitemap.xml

view diff → snapshot

WordPress wordpress.com modified +1 −1 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Corriere della Sera corriere.it modified +0 −10 4mo ago

🤖 AI Unblocked Bytespider (ByteDance) - rules removed
Removed all rules for user-agent ‘Diffbot-User’
Removed all rules for user-agent ‘Scrapy’
🤖 AI Unblocked Timpibot (Timpi) - rules removed
🤖 AI Unblocked YouBot (You.com) - rules removed

view diff → snapshot

Adidas adidas.com modified +1 −1 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Unum Group unum.com modified +8 −1 4mo ago

Added Allow: / for *
Added sitemap https://www.unum.com/sitemap.xml
Removed sitemap https://unum.com/sitemap_unum.xml

view diff → snapshot

WordPress wordpress.com modified +1 −1 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +17 −17 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Pinterest pinterest.com modified +7 −7 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Amgen amgen.com modified +0 −2 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 4mo ago

Added sitemap https://www.bca.co.id/sitemap.xml
Removed sitemap http://www.bca.co.id/sitemap.xml

view diff → snapshot

WordPress wordpress.com modified +1 −1 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +13 −13 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Corriere della Sera corriere.it modified +10 −0 4mo ago

🤖 AI Blocked Bytespider (ByteDance) entirely
Blocked user-agent ‘Diffbot-User’ entirely
Blocked user-agent ‘Scrapy’ entirely
🤖 AI Blocked Timpibot (Timpi) entirely
🤖 AI Blocked YouBot (You.com) entirely

view diff → snapshot

KLA kla.com modified +1 −0 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 4mo ago

Added sitemap http://www.bca.co.id/sitemap.xml
Removed sitemap https://www.bca.co.id/sitemap.xml

view diff → snapshot

WordPress wordpress.com modified +1 −1 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +13 −13 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Etsy etsy.com modified +1 −1 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Glencore glencore.com modified +5 −5 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 4mo ago

Added sitemap http://127.0.0.1/image-sitemap.xml
Added sitemap http://127.0.0.1/page-sitemap.xml
Removed sitemap http://www.iqvia.com/image-sitemap.xml
Removed sitemap http://www.iqvia.com/page-sitemap.xml

view diff → snapshot