The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1198files archived

2679snapshots

1509changes detected

237this week

All changes 🤖 AI-crawler changes Showing initial archives

Temu temu.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

JD.com jd.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

HashiCorp hashicorp.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Coveo coveo.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Mediahuis mediahuis.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

DataDome datadome.co initial 1mo ago

First snapshot of robots.txt archived

snapshot

IQVIA Holdings iqvia.com modified +2 −2 1mo ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

Toronto Star thestar.com modified +18 −18 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

KLA kla.com modified +1 −0 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +16 −16 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

F5 f5.com modified +10 −20 1mo ago

Response is a valid robots.txt again (was an HTML page)
Added rules for user-agent ‘*’
Added sitemap https://www.f5.com/labs/sitemap.xml
Added sitemap https://www.f5.com/sitemap.xml

view diff → snapshot

Advance Auto Parts advanceautoparts.com modified +27 −0 1mo ago

Added rules for user-agent ‘*’
Added rules for user-agent ‘Mediapartners-Google’
Added sitemap https://shop.advanceautoparts.com/sitemap.xml

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 1mo ago

Added sitemap http://www.bca.co.id/sitemap.xml
Removed sitemap https://www.bca.co.id/sitemap.xml

view diff → snapshot

Toronto Star thestar.com modified +13 −13 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

F5 f5.com modified +1 −1 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Pinterest pinterest.com modified +8 −8 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Advance Auto Parts advanceautoparts.com modified +0 −27 1mo ago

Removed all rules for user-agent ‘*’
Removed all rules for user-agent ‘Mediapartners-Google’
Removed sitemap https://shop.advanceautoparts.com/sitemap.xml

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 1mo ago

Added sitemap http://127.0.0.1/image-sitemap.xml
Added sitemap http://127.0.0.1/page-sitemap.xml
Removed sitemap http://www.iqvia.com/image-sitemap.xml
Removed sitemap http://www.iqvia.com/page-sitemap.xml

view diff → snapshot

Mojeek mojeek.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Startpage startpage.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Qwant qwant.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Ecosia ecosia.org initial 1mo ago

First snapshot of robots.txt archived

snapshot

F5 f5.com modified +1 −1 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Poe poe.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Arc Search thebrowser.company initial 1mo ago

First snapshot of robots.txt archived

snapshot

Consensus consensus.app initial 1mo ago

First snapshot of robots.txt archived

snapshot

Webz.io webz.io initial 1mo ago

First snapshot of robots.txt archived

snapshot

Diffbot diffbot.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Common Crawl commoncrawl.org initial 1mo ago

First snapshot of robots.txt archived

snapshot

Schema.org schema.org initial 1mo ago

First snapshot of robots.txt archived

snapshot