The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1206files archived

5433snapshots

4298changes detected

233this week

All changes 🤖 AI-crawler changes Showing initial archives

Toronto Star thestar.com modified +18 −18 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Sysco sysco.com modified +4 −4 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Waldo waldo.fyi modified +2 −2 2mo ago

Added sitemap https://www.waldo.fyi/sitemap.xml
Removed sitemap https://waldo.fyi/sitemap.xml

view diff → snapshot

Figma Community figma.com modified +1 −0 2mo ago

🤖 AI Added Allow: /blog/try-these-10-skills-and-show-off-your-own/$ for CHATGPT-User
🤖 AI Added Allow: /blog/try-these-10-skills-and-show-off-your-own/$ for Claude-SearchBot
🤖 AI Added Allow: /blog/try-these-10-skills-and-show-off-your-own/$ for Claude-User
🤖 AI Added Allow: /blog/try-these-10-skills-and-show-off-your-own/$ for OAI-SearchBot
🤖 AI Added Allow: /blog/try-these-10-skills-and-show-off-your-own/$ for PerplexityBot

view diff → snapshot

Barracuda barracuda.com removed +0 −232 2mo ago

robots.txt is no longer served (HTTP 404)

Bluesky bsky.app modified +1 −0 2mo ago

Added Disallow: /intent/compose for *

view diff → snapshot

CORE core.ac.uk modified +1 −24 2mo ago

Added sitemap https://core.ac.uk/sitemap.xml

view diff → snapshot

Pinterest pinterest.com modified +12 −12 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

X (Twitter) x.com modified +18 −35 2mo ago

Added rules for user-agent ‘meta-externalads’
🤖 AI Added rules for AI crawler meta-externalagent (Meta)
🤖 AI Added rules for AI crawler meta-externalfetcher (Meta)
🤖 AI Added rules for AI crawler Meta-WebIndexer (Meta)

view diff → snapshot

Adidas adidas.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Advance Auto Parts advanceautoparts.com modified +0 −27 2mo ago

Removed all rules for user-agent ‘*’
Removed all rules for user-agent ‘Mediapartners-Google’
Removed sitemap https://shop.advanceautoparts.com/sitemap.xml

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 2mo ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

Gilead Sciences gilead.com modified +5 −5 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 2mo ago

Added sitemap https://www.bca.co.id/sitemap.xml
Removed sitemap http://www.bca.co.id/sitemap.xml

view diff → snapshot

Eli Lilly lilly.com modified +1 −0 2mo ago

Added sitemap https://www.lilly.com/il/sitemap-index.xml

view diff → snapshot

WordPress wordpress.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 2mo ago

Added sitemap http://127.0.0.1/image-sitemap.xml
Added sitemap http://127.0.0.1/page-sitemap.xml
Removed sitemap http://www.iqvia.com/image-sitemap.xml
Removed sitemap http://www.iqvia.com/page-sitemap.xml

view diff → snapshot

DSV dsv.com modified +13 −13 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +18 −18 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

TotalEnergies totalenergies.com modified +1 −1 2mo ago

Added sitemap https://totalenergies.com/sitemap.xml
Removed sitemap https://prd-backoffice.totalenergies.com/sitemap.xml

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 2mo ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

KLA kla.com modified +1 −0 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

DSV dsv.com modified +30 −30 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +17 −17 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

First American Financial firstam.com modified +0 −1 2mo ago

Removed Disallow: /assets/ for *

view diff → snapshot

CBRE Group cbre.com modified +3 −0 2mo ago

Added Disallow: /content/ for *
Added Disallow: /coveo/rest/ for *
Added Disallow: /search-results for *

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 2mo ago

Added sitemap http://127.0.0.1/image-sitemap.xml
Added sitemap http://127.0.0.1/page-sitemap.xml
Removed sitemap http://www.iqvia.com/image-sitemap.xml
Removed sitemap http://www.iqvia.com/page-sitemap.xml

view diff → snapshot

DSV dsv.com modified +28 −28 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot