The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1198files archived

2663snapshots

1493changes detected

221this week

All changes 🤖 AI-crawler changes Showing initial archives

IQVIA Holdings iqvia.com modified +2 −2 1mo ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 1mo ago

Added sitemap https://www.bca.co.id/sitemap.xml
Removed sitemap http://www.bca.co.id/sitemap.xml

view diff → snapshot

Realtor realtor.com modified +3 −0 1mo ago

Blocked user-agent ‘Google-CloudVertexBot’ entirely

view diff → snapshot

WordPress wordpress.com modified +1 −1 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +17 −17 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Indeed indeed.com modified +6 −52 1mo ago

Added Disallow: /*&serpstart= for *
Added Disallow: /*/vaclk for *
Removed Allow: /hire/*?*isid= for *
Removed Allow: /personeel/*?*isid= for *
Removed Allow: /reclutamiento/*?*isid= for *
Removed Allow: /recruiting/*?*isid= for *
… 45 more change(s)

view diff → snapshot

Reichman University runi.ac.il initial 1mo ago

First snapshot of robots.txt archived

snapshot

University of Haifa haifa.ac.il initial 1mo ago

First snapshot of robots.txt archived

snapshot

Bar-Ilan University biu.ac.il initial 1mo ago

First snapshot of robots.txt archived

snapshot

Ben-Gurion University of the Negev bgu.ac.il initial 1mo ago

First snapshot of robots.txt archived

snapshot

Weizmann Institute of Science weizmann.ac.il initial 1mo ago

First snapshot of robots.txt archived

snapshot

Technion int.technion.ac.il initial 1mo ago

First snapshot of robots.txt archived

snapshot

BBC bbc.com modified +1 −1 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Barracuda barracuda.com removed +0 −232 1mo ago

robots.txt is no longer served (HTTP 404)

Recorded Future recordedfuture.com modified +1 −1 1mo ago

Added Disallow: /cdn-cgi/ for *

view diff → snapshot

Etsy etsy.com modified +1 −1 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Pinterest pinterest.com modified +7 −7 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Adidas adidas.com modified +1 −1 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +15 −15 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

McGill University mcgill.ca initial 1mo ago

First snapshot of robots.txt archived

snapshot

Etsy etsy.com modified +1 −1 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

ArcelorMittal arcelormittal.com modified +0 −4 1mo ago

Removed sitemap https://corporate.arcelormittal.com/sitemap.xml

view diff → snapshot

University of Cape Town uct.ac.za initial 1mo ago

First snapshot of robots.txt archived

snapshot

Tel Aviv University tau.ac.il initial 1mo ago

First snapshot of robots.txt archived

snapshot

University of British Columbia ubc.ca initial 1mo ago

First snapshot of robots.txt archived

snapshot

University of Toronto utoronto.ca initial 1mo ago

First snapshot of robots.txt archived

snapshot

Indian Institute of Science iisc.ac.in initial 1mo ago

First snapshot of robots.txt archived

snapshot

Indian Institute of Technology Bombay iitb.ac.in initial 1mo ago

First snapshot of robots.txt archived

snapshot

University of Sydney sydney.edu.au initial 1mo ago

First snapshot of robots.txt archived

snapshot