The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1206files archived

5436snapshots

4301changes detected

236this week

All changes 🤖 AI-crawler changes Hiding initial archives

SoundCloud soundcloud.com modified +60 −29 3mo ago

🤖 AI Blocked Amazonbot (Amazon) entirely
🤖 AI Blocked Applebot-Extended (Apple) entirely
🤖 AI Blocked Bytespider (ByteDance) entirely
🤖 AI Blocked ChatGPT-User (OpenAI) entirely
🤖 AI Blocked Claude-Web (Anthropic) entirely
🤖 AI Blocked cohere-ai (Cohere) entirely
… 45 more change(s)

view diff → snapshot

Tractor Supply tractorsupply.com modified +12 −0 3mo ago

Added Disallow: /gtwy/api/accountservice/orchestration/accounts/change/password for *
Added Disallow: /gtwy/api/accountservice/orchestration/accounts/forgot/password for *
Added Disallow: /gtwy/api/accountservice/orchestration/accounts/lookup for *
Added Disallow: /gtwy/api/accountservice/orchestration/accounts/update for *
Added Disallow: /gtwy/api/accountservice/orchestration/accounts/users/ncenroll for *
Added Disallow: /gtwy/api/accountservice/orchestration/accounts/verifyTicket for *
… 6 more change(s)

view diff → snapshot

Owens Corning owenscorning.com modified +1 −32 3mo ago

Removed all rules for user-agent ‘*’
Removed sitemap https://www.owenscorning.com/sitemap-1.xml
Removed sitemap https://www.owenscorning.com/sitemap.xml

view diff → snapshot

DSV dsv.com modified +23 −23 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Amgen amgen.com modified +0 −2 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Eli Lilly lilly.com modified +1 −0 3mo ago

Added sitemap https://www.lilly.com/br/sitemap-index.xml

view diff → snapshot

WordPress wordpress.com modified +1 −1 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Societe Generale societegenerale.com removed +0 −63 3mo ago

robots.txt is no longer served (HTTP 404)

Toronto Star thestar.com modified +13 −13 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Barracuda barracuda.com created +28 −28 3mo ago

Added Disallow: /70hkWk5EPQLRDbV9Q7189DJB_y14lTx9BS0qLGJVSZA=.html for *
Removed Disallow: /2QGvuWDJzNTl_zQtUa-kCYxc421SSSioP4rLS4V3Tic=.html for *
Added Disallow: /70hkWk5EPQLRDbV9Q7189DJB_y14lTx9BS0qLGJVSZA=.html for DotBot
Removed Disallow: /2QGvuWDJzNTl_zQtUa-kCYxc421SSSioP4rLS4V3Tic=.html for DotBot
Added Disallow: /70hkWk5EPQLRDbV9Q7189DJB_y14lTx9BS0qLGJVSZA=.html for sistrix
Removed Disallow: /2QGvuWDJzNTl_zQtUa-kCYxc421SSSioP4rLS4V3Tic=.html for sistrix
… 2 more change(s)

view diff → snapshot

Owens Corning owenscorning.com modified +32 −1 3mo ago

Added rules for user-agent ‘*’
Added sitemap https://www.owenscorning.com/sitemap-1.xml
Added sitemap https://www.owenscorning.com/sitemap.xml

view diff → snapshot

DSV dsv.com modified +25 −25 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +14 −14 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Barracuda barracuda.com removed +0 −232 3mo ago

robots.txt is no longer served (HTTP 404)

L'Oreal loreal.com modified +3 −0 3mo ago

Added sitemap https://www.loreal.com/sitemap-image.xml
Added sitemap https://www.loreal.com/sitemap.xml

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 3mo ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

KLA kla.com modified +1 −0 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

DSV dsv.com modified +27 −27 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 3mo ago

Added sitemap http://www.bca.co.id/sitemap.xml
Removed sitemap https://www.bca.co.id/sitemap.xml

view diff → snapshot

Kraken kraken.com modified +2 −2 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +14 −14 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

L'Oreal loreal.com modified +0 −3 3mo ago

Removed sitemap https://www.loreal.com/sitemap-image.xml
Removed sitemap https://www.loreal.com/sitemap.xml

view diff → snapshot

Roblox roblox.com modified +0 −61 3mo ago

🤖 AI Unblocked Amazonbot (Amazon) - rules removed
🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
🤖 AI Unblocked Bytespider (ByteDance) - rules removed
🤖 AI Unblocked CCBot (Common Crawl) - rules removed
🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
… 4 more change(s)

view diff → snapshot

Forbes forbes.com modified +2 −0 3mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

ABM Industries abm.com modified +1 −1 3mo ago

Added Disallow: /project-docs/ai-agent-training for *
Removed Disallow: /ai-agent-training for *

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 3mo ago

Added sitemap http://127.0.0.1/image-sitemap.xml
Added sitemap http://127.0.0.1/page-sitemap.xml
Removed sitemap http://www.iqvia.com/image-sitemap.xml
Removed sitemap http://www.iqvia.com/page-sitemap.xml

view diff → snapshot

General Mills generalmills.com modified +2 −9 3mo ago

Added Allow: / for *
Removed Disallow: /*.aspx for *
Removed Disallow: /*p=1 for *
Removed Disallow: /.well-known/apple-app-site-association for *
Removed Disallow: /apple-app-site-association for *
Removed Disallow: /search for *
… 4 more change(s)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 3mo ago

Added sitemap https://www.bca.co.id/sitemap.xml
Removed sitemap http://www.bca.co.id/sitemap.xml

view diff → snapshot