NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
Proofpoint proofpoint.com modified +9 −0
  • Added Disallow: /sites/default/files/e-books/pfpt-au-eb-buyers-guide-choosing-dspm-solution-ai-era.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-br-eb-buyers-guide-choosing-dspm-solution-ai-era.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-de-eb-buyers-guide-choosing-dspm-solution-ai-era.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-es-eb-buyers-guide-choosing-dspm-solution-ai-era.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-fr-eb-buyers-guide-choosing-dspm-solution-ai-era.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-it-eb-buyers-guide-choosing-dspm-solution-ai-era.pdf for *
  • … 3 more change(s)
view diff → snapshot
Target target.com removed +0 −122
  • robots.txt is no longer served (HTTP 404)
Lithia Motors lithiamotors.com modified +70 −1
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • Blocked user-agent ‘atlassian-bot’ entirely
  • Blocked user-agent ‘AwarioRssBot’ entirely
  • Blocked user-agent ‘AwarioSmartBot’ entirely
  • Blocked user-agent ‘Baiduspider’ entirely
  • Blocked user-agent ‘BorderxBot’ entirely
  • … 19 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Expeditors International expeditors.com modified +130 −0
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Blocked user-agent ‘atlassian-bot’ entirely
  • Blocked user-agent ‘AwarioRssBot’ entirely
  • Blocked user-agent ‘AwarioSmartBot’ entirely
  • … 27 more change(s)
view diff → snapshot
Instagram instagram.com modified +18 −66
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Removed Allow: /places/c/ for Amazonbot
  • 🤖 AI Removed Disallow: /accounts/login/*?next= for Amazonbot
  • 🤖 AI Removed Disallow: /ajax/ for Amazonbot
  • 🤖 AI Removed Disallow: /client_error/ for Amazonbot
  • 🤖 AI Removed Disallow: /direct/ for Amazonbot
  • … 45 more change(s)
view diff → snapshot
Facebook facebook.com modified +18 −282
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Removed Allow: /ajax/bootloader-endpoint/ for Amazonbot
  • 🤖 AI Removed Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet for Amazonbot
  • 🤖 AI Removed Allow: /careers/ for Amazonbot
  • 🤖 AI Removed Allow: /safetycheck/ for Amazonbot
  • 🤖 AI Removed Disallow: /*/plugins/* for Amazonbot
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot