The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1206files archived

5439snapshots

4304changes detected

239this week

All changes 🤖 AI-crawler changes Showing initial archives

Bank Central Asia bca.co.id modified +1 −1 4mo ago

Added sitemap https://www.bca.co.id/sitemap.xml
Removed sitemap http://www.bca.co.id/sitemap.xml

view diff → snapshot

Realtor realtor.com modified +3 −0 4mo ago

Blocked user-agent ‘Google-CloudVertexBot’ entirely

view diff → snapshot

WordPress wordpress.com modified +1 −1 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +17 −17 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Indeed indeed.com modified +6 −52 4mo ago

Added Disallow: /*&serpstart= for *
Added Disallow: /*/vaclk for *
Removed Allow: /hire/*?*isid= for *
Removed Allow: /personeel/*?*isid= for *
Removed Allow: /reclutamiento/*?*isid= for *
Removed Allow: /recruiting/*?*isid= for *
… 45 more change(s)

view diff → snapshot

Reichman University runi.ac.il initial 4mo ago

First snapshot of robots.txt archived

snapshot

University of Haifa haifa.ac.il initial 4mo ago

First snapshot of robots.txt archived

snapshot

Bar-Ilan University biu.ac.il initial 4mo ago

First snapshot of robots.txt archived

snapshot

Ben-Gurion University of the Negev bgu.ac.il initial 4mo ago

First snapshot of robots.txt archived

snapshot

Weizmann Institute of Science weizmann.ac.il initial 4mo ago

First snapshot of robots.txt archived

snapshot

Technion int.technion.ac.il initial 4mo ago

First snapshot of robots.txt archived

snapshot

BBC bbc.com modified +1 −1 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Barracuda barracuda.com removed +0 −232 4mo ago

robots.txt is no longer served (HTTP 404)

Recorded Future recordedfuture.com modified +1 −1 4mo ago

Added Disallow: /cdn-cgi/ for *

view diff → snapshot

Etsy etsy.com modified +1 −1 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Pinterest pinterest.com modified +7 −7 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Adidas adidas.com modified +1 −1 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +15 −15 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

McGill University mcgill.ca initial 4mo ago

First snapshot of robots.txt archived

snapshot

Etsy etsy.com modified +1 −1 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

ArcelorMittal arcelormittal.com modified +0 −4 4mo ago

Removed sitemap https://corporate.arcelormittal.com/sitemap.xml

view diff → snapshot

University of Cape Town uct.ac.za initial 4mo ago

First snapshot of robots.txt archived

snapshot

Tel Aviv University tau.ac.il initial 4mo ago

First snapshot of robots.txt archived

snapshot

University of British Columbia ubc.ca initial 4mo ago

First snapshot of robots.txt archived

snapshot

University of Toronto utoronto.ca initial 4mo ago

First snapshot of robots.txt archived

snapshot

Indian Institute of Science iisc.ac.in initial 4mo ago

First snapshot of robots.txt archived

snapshot

Indian Institute of Technology Bombay iitb.ac.in initial 4mo ago

First snapshot of robots.txt archived

snapshot

University of Sydney sydney.edu.au initial 4mo ago

First snapshot of robots.txt archived

snapshot

Nanyang Technological University ntu.edu.sg initial 4mo ago

First snapshot of robots.txt archived

snapshot