NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Showing initial archives
University of Cape Town uct.ac.za modified +1 −1
  • Added sitemap https://uct.ac.za/sitemap.xmlThe website encountered an unexpected error. Try again later.<br />
  • Removed sitemap https://uct.ac.za/sitemap.xml
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +29 −2
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /marketplace/az/* for *
  • Added Disallow: /marketplace/ca/* for *
  • Added Disallow: /marketplace/fl/* for *
  • Added Disallow: /marketplace/ga/* for *
  • Added Disallow: /marketplace/ia/* for *
  • … 23 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Guyana (Office of the President) op.gov.gy modified +130 −0
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Blocked user-agent ‘atlassian-bot’ entirely
  • Blocked user-agent ‘AwarioRssBot’ entirely
  • Blocked user-agent ‘AwarioSmartBot’ entirely
  • … 27 more change(s)
view diff → snapshot
StoneX Group stonex.com modified +6 −0
  • Added Disallow: /*.aspx for *
  • Added Disallow: /*.map for *
  • Added Disallow: /*.php for *
  • Added Disallow: /*~/ for *
  • Added Disallow: /api/barchart-widget/ for *
  • Added Disallow: /layouts/ for *
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot