The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1198files archived

2679snapshots

1509changes detected

221this week

All changes 🤖 AI-crawler changes Showing initial archives

Moz moz.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Glencore glencore.com modified +5 −5 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Similarweb similarweb.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Ahrefs ahrefs.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Semrush semrush.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Toyota toyota.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Adidas adidas.com modified +1 −1 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Philip Morris International pmi.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Goodyear Tire & Rubber goodyear.com modified +40 −0 1mo ago

Blocked user-agent ‘ahrefsbot’ entirely
Blocked user-agent ‘cazoodlebot’ entirely
Blocked user-agent ‘dotbot/1.0’ entirely
Blocked user-agent ‘gigabot’ entirely
Blocked user-agent ‘mj12bot’ entirely
Blocked user-agent ‘sogou inst spider’ entirely
… 20 more change(s)

view diff → snapshot

Kweichow Moutai moutaichina.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Zhongji Innolight innolight.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

KIOXIA kioxia.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Amgen amgen.com modified +2 −0 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 1mo ago

Added sitemap https://www.bca.co.id/sitemap.xml
Removed sitemap http://www.bca.co.id/sitemap.xml

view diff → snapshot

Palantir palantir.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Arm Holdings arm.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Hon Hai (Foxconn) honhai.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Toronto Star thestar.com modified +15 −15 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

ExtraHop extrahop.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Vectra AI vectra.ai initial 1mo ago

First snapshot of robots.txt archived

snapshot

McAfee mcafee.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Avast avast.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Arkose Labs arkoselabs.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Kasada kasada.io initial 1mo ago

First snapshot of robots.txt archived

snapshot

F5 f5.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Imperva imperva.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Malwarebytes malwarebytes.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Human Security humansecurity.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Illumio illumio.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Kaspersky kaspersky.com initial 1mo ago

First snapshot of robots.txt archived

snapshot