The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1206files archived

5433snapshots

4298changes detected

233this week

All changes 🤖 AI-crawler changes Hiding initial archives

Bitdefender bitdefender.com modified +1 −0 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Fox News foxnews.com modified +3 −1 2mo ago

Added Disallow: /api/article-search for *
Added Disallow: /xid for *
Removed sitemap https://video.foxnews.com/videositemap.xml

view diff → snapshot

Semrush semrush.com modified +1 −0 2mo ago

Added Allow: /analytics/keywordoverview/*?q=* for *

view diff → snapshot

Tata Consultancy Services tcs.com modified +2 −0 2mo ago

Added Disallow: /content/dam/global-tcs/en/pdfs/what-we-do/platforms/add/add-analytics-insight-solution.pdf for *
Added Disallow: /content/dam/global-tcs/en/pdfs/what-we-do/platforms/tcs-add/solution/tcs-add-analytics-insights-intelligent-decision-making.pdf for *

view diff → snapshot

DSV dsv.com modified +25 −25 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Renault Group renaultgroup.com modified +0 −72 2mo ago

🤖 AI Unblocked ClaudeBot (Anthropic)
🤖 AI Removed Allow: /br/ for ClaudeBot
🤖 AI Removed Allow: /de/ for ClaudeBot
🤖 AI Removed Allow: /es/ for ClaudeBot
🤖 AI Removed Allow: /it/ for ClaudeBot
🤖 AI Removed Allow: /pt/ for ClaudeBot
… 8 more change(s)

view diff → snapshot

WordPress wordpress.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +14 −14 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

DOAJ doaj.org modified +20 −68 2mo ago

🤖 AI Unblocked Amazonbot (Amazon) - rules removed
🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
🤖 AI Unblocked Bytespider (ByteDance) - rules removed
🤖 AI Unblocked CCBot (Common Crawl) - rules removed
🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
… 21 more change(s)

view diff → snapshot

Fox News foxnews.com modified +1 −3 2mo ago

Removed Disallow: /api/article-search for *
Removed Disallow: /xid for *
Added sitemap https://video.foxnews.com/videositemap.xml

view diff → snapshot

KLA kla.com modified +1 −0 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

DSV dsv.com modified +20 −20 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Amgen amgen.com modified +2 −0 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +16 −16 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Indeed indeed.com modified +30 −0 2mo ago

Added Allow: /*&start=0& for *
Added Allow: /*&start=10& for *
Added Allow: /*&start=20& for *
Added Allow: /*&start=30& for *
Added Allow: /*&start=40& for *
Added Allow: /*&start=50& for *
… 45 more change(s)

view diff → snapshot

Etsy etsy.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

eBay ebay.com modified +5 −1 2mo ago

Added Disallow: /d/graphql for *
Added Disallow: /looking-glass-proxy/graphql for *
Added Disallow: /d/graphql for bingbot
Added Disallow: /looking-glass-proxy/graphql for bingbot

view diff → snapshot

Advance Auto Parts advanceautoparts.com modified +27 −0 2mo ago

Added rules for user-agent ‘*’
Added rules for user-agent ‘Mediapartners-Google’
Added sitemap https://shop.advanceautoparts.com/sitemap.xml

view diff → snapshot

DSV dsv.com modified +28 −28 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Amgen amgen.com modified +0 −2 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Nubank nubank.com.br modified +32 −30 2mo ago

Added Disallow: /*&utm_ for *
Added Disallow: /*?utm_ for *

view diff → snapshot

WordPress wordpress.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +14 −14 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

The Wall Street Journal wsj.com modified +1 −0 2mo ago

Added rules for user-agent ‘Wallie’

view diff → snapshot

TotalEnergies totalenergies.com modified +1 −1 2mo ago

Added sitemap https://totalenergies.com/sitemap.xml
Removed sitemap https://prd-backoffice.totalenergies.com/sitemap.xml

view diff → snapshot

Adidas adidas.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Advance Auto Parts advanceautoparts.com modified +0 −27 2mo ago

Removed all rules for user-agent ‘*’
Removed all rules for user-agent ‘Mediapartners-Google’
Removed sitemap https://shop.advanceautoparts.com/sitemap.xml

view diff → snapshot

DXC Technology dxc.com modified +2 −0 2mo ago

Added Disallow: */language-masters/ for *
Added sitemap https://dxc.com/newsroom-sitemap.xml

view diff → snapshot

DSV dsv.com modified +24 −24 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot