The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1206files archived

5439snapshots

4304changes detected

239this week

All changes 🤖 AI-crawler changes Showing initial archives

Germany (Bundesregierung) bundesregierung.de initial 4mo ago

First snapshot of robots.txt archived

snapshot

Booking.com booking.com initial 4mo ago

First snapshot of robots.txt archived

snapshot

BlackRock blackrock.com initial 4mo ago

First snapshot of robots.txt archived

snapshot

ArcelorMittal arcelormittal.com initial 4mo ago

First snapshot of robots.txt archived

snapshot

Allianz allianz.com initial 4mo ago

First snapshot of robots.txt archived

snapshot

Goodyear Tire & Rubber goodyear.com modified +0 −40 4mo ago

Removed all rules for user-agent ‘ahrefsbot’
Removed all rules for user-agent ‘cazoodlebot’
Removed all rules for user-agent ‘dotbot/1.0’
Removed all rules for user-agent ‘gigabot’
Removed all rules for user-agent ‘mj12bot’
Removed all rules for user-agent ‘sogou inst spider’
… 20 more change(s)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 4mo ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

Amgen amgen.com modified +0 −2 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +15 −15 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Alamy alamy.com initial 4mo ago

First snapshot of robots.txt archived

snapshot

Bank Central Asia bca.co.id modified +1 −1 4mo ago

Added sitemap http://www.bca.co.id/sitemap.xml
Removed sitemap https://www.bca.co.id/sitemap.xml

view diff → snapshot

Toronto Star thestar.com modified +15 −15 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Glencore glencore.com modified +5 −5 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

GXO Logistics gxo.com initial 4mo ago

First snapshot of robots.txt archived

snapshot

Toronto Star thestar.com modified +16 −16 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Glencore glencore.com modified +5 −5 4mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Khan Academy khanacademy.org initial 4mo ago

First snapshot of robots.txt archived

snapshot

Instructables instructables.com initial 4mo ago

First snapshot of robots.txt archived

snapshot

Letterboxd letterboxd.com initial 4mo ago

First snapshot of robots.txt archived

snapshot

PyPI pypi.org initial 4mo ago

First snapshot of robots.txt archived

snapshot

Read the Docs readthedocs.org initial 4mo ago

First snapshot of robots.txt archived

snapshot

Project Gutenberg gutenberg.org initial 4mo ago

First snapshot of robots.txt archived

snapshot

Internet Archive archive.org initial 4mo ago

First snapshot of robots.txt archived

snapshot

Wiktionary wiktionary.org initial 4mo ago

First snapshot of robots.txt archived

snapshot

Wikimedia Commons commons.wikimedia.org initial 4mo ago

First snapshot of robots.txt archived

snapshot

Hacker News news.ycombinator.com initial 4mo ago

First snapshot of robots.txt archived

snapshot

DEV Community dev.to initial 4mo ago

First snapshot of robots.txt archived

snapshot

Goodreads goodreads.com initial 4mo ago

First snapshot of robots.txt archived

snapshot

wikiHow wikihow.com initial 4mo ago

First snapshot of robots.txt archived

snapshot

GitLab gitlab.com initial 4mo ago

First snapshot of robots.txt archived

snapshot