The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1206files archived

5433snapshots

4298changes detected

233this week

All changes 🤖 AI-crawler changes Showing initial archives

Fox News foxnews.com modified +1 −3 2mo ago

Removed Disallow: /api/article-search for *
Removed Disallow: /xid for *
Added sitemap https://video.foxnews.com/videositemap.xml

view diff → snapshot

Pinterest pinterest.com modified +13 −13 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

The New York Times nytimes.com modified +15 −0 2mo ago

Blocked user-agent ‘Brightbot’ entirely
Blocked user-agent ‘Diffbot-User’ entirely
Blocked user-agent ‘ExaBot’ entirely
Blocked user-agent ‘FirecrawlAgent’ entirely
Blocked user-agent ‘ShapBot’ entirely

view diff → snapshot

Adidas adidas.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 2mo ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

DSV dsv.com modified +25 −25 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Gilead Sciences gilead.com modified +4 −4 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Tripadvisor tripadvisor.com modified +3 −28 2mo ago

Added Disallow: /BookingConfirmationSMSResponse for *
Added Disallow: /BookingConfirmationShare for *
Added Disallow: /TripX for *
Removed Disallow: /BookingConfirmation for *
Removed Disallow: /CheapFlightsCommerce for *
Removed Disallow: /CheapFlightsSearchDeals for *
… 45 more change(s)

view diff → snapshot

WordPress wordpress.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +16 −16 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bild bild.de modified +1 −0 2mo ago

🤖 AI Blocked Amazonbot (Amazon) entirely

view diff → snapshot

BBC bbc.com modified +17 −13 2mo ago

Added sitemap https://www.bbc.com/sitemaps/https-index-com-mundo-archive.xml
Added sitemap https://www.bbc.com/sitemaps/https-index-com-mundo-archive_video.xml
Added sitemap https://www.bbc.com/sitemaps/https-index-com-mundo-news.xml
Added sitemap https://www.bbc.com/sitemaps/https-index-com-mundo-video.xml

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 2mo ago

Added sitemap http://127.0.0.1/image-sitemap.xml
Added sitemap http://127.0.0.1/page-sitemap.xml
Removed sitemap http://www.iqvia.com/image-sitemap.xml
Removed sitemap http://www.iqvia.com/page-sitemap.xml

view diff → snapshot

Gilead Sciences gilead.com modified +5 −5 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +15 −15 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 2mo ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

KLA kla.com modified +1 −0 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

DSV dsv.com modified +26 −26 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +15 −15 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Etsy etsy.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 2mo ago

Added sitemap http://127.0.0.1/image-sitemap.xml
Added sitemap http://127.0.0.1/page-sitemap.xml
Removed sitemap http://www.iqvia.com/image-sitemap.xml
Removed sitemap http://www.iqvia.com/page-sitemap.xml

view diff → snapshot

DSV dsv.com modified +27 −27 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Amgen amgen.com modified +2 −0 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Pinterest pinterest.com modified +9 −9 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Adidas adidas.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 2mo ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

DSV dsv.com modified +22 −22 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Amgen amgen.com modified +0 −2 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot