- Added rules for user-agent ‘AndiBot’
- 🤖 AI Blocked anthropic-ai (Anthropic) entirely
- Added rules for user-agent ‘Applebot’
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- Added rules for user-agent ‘Bingbot’
- 🤖 AI Blocked Bytespider (ByteDance) entirely
- … 45 more change(s)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.lilly.com/at/sitemap-index.xml
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /api/* for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /ausland/ for Claude-SearchBot
- 🤖 AI Added Allow: /familie/ for Claude-SearchBot
- 🤖 AI Added Allow: /gesundheit/ for Claude-SearchBot
- 🤖 AI Added Allow: /partnerschaft/ for Claude-SearchBot
- 🤖 AI Added Allow: /psychologie/ for Claude-SearchBot
- 🤖 AI Added Allow: /reise/ for Claude-SearchBot
- … 18 more change(s)
- Removed Allow: /*/ar/4009?appliedRefinements=2828 for *
- Removed Allow: /en-ph/*/sc/ for *
- Removed Allow: /en-vn/*/sc/ for *
- Removed Disallow: /support/kbdoc/*/article/lkbprint for *
- 🤖 AI Added Allow: /blog/london-office-expansion/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/london-office-expansion/$ for Claude-SearchBot
- 🤖 AI Added Allow: /blog/london-office-expansion/$ for Claude-User
- 🤖 AI Added Allow: /blog/london-office-expansion/$ for OAI-SearchBot
- 🤖 AI Added Allow: /blog/london-office-expansion/$ for PerplexityBot
- Blocked user-agent ‘ahrefsbot’ entirely
- Blocked user-agent ‘cazoodlebot’ entirely
- Blocked user-agent ‘dotbot/1.0’ entirely
- Blocked user-agent ‘gigabot’ entirely
- Blocked user-agent ‘mj12bot’ entirely
- Blocked user-agent ‘sogou inst spider’ entirely
- … 20 more change(s)
- Removed all rules for user-agent ‘AndiBot’
- 🤖 AI Unblocked anthropic-ai (Anthropic) - rules removed
- Removed all rules for user-agent ‘Applebot’
- 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
- Removed all rules for user-agent ‘Bingbot’
- 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘ahrefsbot’
- Removed all rules for user-agent ‘cazoodlebot’
- Removed all rules for user-agent ‘dotbot/1.0’
- Removed all rules for user-agent ‘gigabot’
- Removed all rules for user-agent ‘mj12bot’
- Removed all rules for user-agent ‘sogou inst spider’
- … 20 more change(s)
- Added rules for user-agent ‘*’
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- 🤖 AI Blocked Bytespider (ByteDance) entirely
- 🤖 AI Blocked CCBot (Common Crawl) entirely
- 🤖 AI Blocked ClaudeBot (Anthropic) entirely
- 🤖 AI Blocked Google-Extended (Google) entirely
- … 4 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /sites/default/files/e-books/pfpt-au-eb-the-new-rules-of-business-communication-governance.pdf for *
- Added Disallow: /sites/default/files/e-books/pfpt-br-eb-the-new-rules-of-business-communication-governance.pdf for *
- Added Disallow: /sites/default/files/e-books/pfpt-de-eb-the-new-rules-of-business-communication-governance.pdf for *
- Added Disallow: /sites/default/files/e-books/pfpt-es-eb-the-new-rules-of-business-communication-governance.pdf for *
- Added Disallow: /sites/default/files/e-books/pfpt-fr-eb-the-new-rules-of-business-communication-governance.pdf for *
- Added Disallow: /sites/default/files/e-books/pfpt-it-eb-the-new-rules-of-business-communication-governance.pdf for *
- … 1 more change(s)
- Added rules for user-agent ‘NewsBank.com’
- Added Allow: /story/branded-story/ for ChatGPT Operator
- Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT Operator
- Added Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT Operator
- 🤖 AI Added Allow: /story/branded-story/ for ChatGPT-User
- 🤖 AI Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT-User
- … 13 more change(s)
- Added Disallow: /cl/* for *
- Added Disallow: /cl/* for AdIdxBot
- Added Disallow: /cl/* for AdsBot-Google
- Added Disallow: /cl/* for AdsBot-Google-Mobile
- 🤖 AI Added Disallow: /cl/* for ChatGPT-User
- 🤖 AI Added Disallow: /cl/* for Claude-SearchBot
- … 7 more change(s)
- Removed all rules for user-agent ‘*’
- 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
- 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
- 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
- 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
- 🤖 AI Unblocked Google-Extended (Google) - rules removed
- … 4 more change(s)
- Removed sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-ca-es-sitemap_index.xml
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- Added Disallow: /alice*? for *
- Removed Disallow: /alice/*? for *
- Added Disallow: /alice*? for Bingbot
- Removed Disallow: /alice/*? for Bingbot
- Added Disallow: /alice*? for Googlebot
- Removed Disallow: /alice/*? for Googlebot
- … 2 more change(s)