NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
All changes 🤖 AI-crawler changes Hiding initial archives
Der Spiegel spiegel.de modified +36 −6
  • 🤖 AI Added Allow: /ausland/ for Claude-SearchBot
  • 🤖 AI Added Allow: /familie/ for Claude-SearchBot
  • 🤖 AI Added Allow: /gesundheit/ for Claude-SearchBot
  • 🤖 AI Added Allow: /partnerschaft/ for Claude-SearchBot
  • 🤖 AI Added Allow: /psychologie/ for Claude-SearchBot
  • 🤖 AI Added Allow: /reise/ for Claude-SearchBot
  • … 18 more change(s)
view diff → snapshot
Dell Technologies dell.com modified +1 −5
  • Removed Allow: /*/ar/4009?appliedRefinements=2828 for *
  • Removed Allow: /en-ph/*/sc/ for *
  • Removed Allow: /en-vn/*/sc/ for *
  • Removed Disallow: /support/kbdoc/*/article/lkbprint for *
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/london-office-expansion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/london-office-expansion/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/london-office-expansion/$ for Claude-User
  • 🤖 AI Added Allow: /blog/london-office-expansion/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/london-office-expansion/$ for PerplexityBot
view diff → snapshot
Goodyear Tire & Rubber goodyear.com modified +40 −0
  • Blocked user-agent ‘ahrefsbot’ entirely
  • Blocked user-agent ‘cazoodlebot’ entirely
  • Blocked user-agent ‘dotbot/1.0’ entirely
  • Blocked user-agent ‘gigabot’ entirely
  • Blocked user-agent ‘mj12bot’ entirely
  • Blocked user-agent ‘sogou inst spider’ entirely
  • … 20 more change(s)
view diff → snapshot
Kimberly-Clark kimberly-clark.com modified +2 −109
  • Removed all rules for user-agent ‘AndiBot’
  • 🤖 AI Unblocked anthropic-ai (Anthropic) - rules removed
  • Removed all rules for user-agent ‘Applebot’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • Removed all rules for user-agent ‘Bingbot’
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • … 45 more change(s)
view diff → snapshot
Goodyear Tire & Rubber goodyear.com modified +0 −40
  • Removed all rules for user-agent ‘ahrefsbot’
  • Removed all rules for user-agent ‘cazoodlebot’
  • Removed all rules for user-agent ‘dotbot/1.0’
  • Removed all rules for user-agent ‘gigabot’
  • Removed all rules for user-agent ‘mj12bot’
  • Removed all rules for user-agent ‘sogou inst spider’
  • … 20 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Proofpoint proofpoint.com modified +7 −0
  • Added Disallow: /sites/default/files/e-books/pfpt-au-eb-the-new-rules-of-business-communication-governance.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-br-eb-the-new-rules-of-business-communication-governance.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-de-eb-the-new-rules-of-business-communication-governance.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-es-eb-the-new-rules-of-business-communication-governance.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-fr-eb-the-new-rules-of-business-communication-governance.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-it-eb-the-new-rules-of-business-communication-governance.pdf for *
  • … 1 more change(s)
view diff → snapshot
USA Today usatoday.com modified +27 −6
  • Added rules for user-agent ‘NewsBank.com’
  • Added Allow: /story/branded-story/ for ChatGPT Operator
  • Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT Operator
  • Added Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT Operator
  • 🤖 AI Added Allow: /story/branded-story/ for ChatGPT-User
  • 🤖 AI Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT-User
  • … 13 more change(s)
view diff → snapshot
Expedia Group expedia.com modified +6 −0
  • Added Disallow: /cl/* for *
  • Added Disallow: /cl/* for AdIdxBot
  • Added Disallow: /cl/* for AdsBot-Google
  • Added Disallow: /cl/* for AdsBot-Google-Mobile
  • 🤖 AI Added Disallow: /cl/* for ChatGPT-User
  • 🤖 AI Added Disallow: /cl/* for Claude-SearchBot
  • … 7 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
DSV dsv.com modified +24 −25
  • Removed sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-ca-es-sitemap_index.xml
view diff → snapshot
Arrow Electronics arrow.com removed +0 −102
  • robots.txt is no longer served (HTTP 404)
Yandex yandex.ru modified +4 −4
  • Added Disallow: /alice*? for *
  • Removed Disallow: /alice/*? for *
  • Added Disallow: /alice*? for Bingbot
  • Removed Disallow: /alice/*? for Bingbot
  • Added Disallow: /alice*? for Googlebot
  • Removed Disallow: /alice/*? for Googlebot
  • … 2 more change(s)
view diff → snapshot