The Pulse

Every change to a monitored robots.txt, archived and explained.

1166sites tracked

1052files archived

1071snapshots

19changes detected

19this week

All changes 🤖 AI-crawler changes Showing initial archives

Clearscope clearscope.io initial 5h ago

First snapshot of robots.txt archived

snapshot

Surfer surferseo.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Lumar lumar.io initial 5h ago

First snapshot of robots.txt archived

snapshot

Conductor conductor.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Botify botify.com initial 5h ago

First snapshot of robots.txt archived

snapshot

seoClarity seoclarity.net initial 5h ago

First snapshot of robots.txt archived

snapshot

You.com you.com initial 5h ago

First snapshot of robots.txt archived

snapshot

DuckDuckGo duckduckgo.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Brave Search brave.com initial 5h ago

First snapshot of robots.txt archived

snapshot

BrightEdge brightedge.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Microsoft Bing bing.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Google google.com initial 5h ago

First snapshot of robots.txt archived

snapshot

NeuralCrawl neuralcrawl.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Profound tryprofound.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Moz moz.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Glencore glencore.com modified +5 −5 5h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Similarweb similarweb.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Ahrefs ahrefs.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Semrush semrush.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Toyota toyota.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Adidas adidas.com modified +1 −1 5h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Philip Morris International pmi.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Goodyear Tire & Rubber goodyear.com modified +40 −0 5h ago

Blocked user-agent ‘ahrefsbot’ entirely
Blocked user-agent ‘cazoodlebot’ entirely
Blocked user-agent ‘dotbot/1.0’ entirely
Blocked user-agent ‘gigabot’ entirely
Blocked user-agent ‘mj12bot’ entirely
Blocked user-agent ‘sogou inst spider’ entirely
… 20 more change(s)

view diff → snapshot

Kweichow Moutai moutaichina.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Zhongji Innolight innolight.com initial 5h ago

First snapshot of robots.txt archived

snapshot

KIOXIA kioxia.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Amgen amgen.com modified +2 −0 5h ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 5h ago

Added sitemap https://www.bca.co.id/sitemap.xml
Removed sitemap http://www.bca.co.id/sitemap.xml

view diff → snapshot

Palantir palantir.com initial 5h ago

First snapshot of robots.txt archived

snapshot

Arm Holdings arm.com initial 5h ago

First snapshot of robots.txt archived

snapshot