NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2609snapshots
1437changes detected
226this week
All changes 🤖 AI-crawler changes Showing initial archives
Roblox roblox.com modified +61 −0
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • Blocked user-agent ‘CloudflareBrowserRenderingCrawler’ entirely
  • … 4 more change(s)
view diff → snapshot
Weibo weibo.com modified +15 −35
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • Removed all rules for user-agent ‘ChatGLM’
  • 🤖 AI Unblocked ChatGPT-User (OpenAI) - rules removed
  • Removed all rules for user-agent ‘DeepSeek’
  • Removed all rules for user-agent ‘DeepSeek Chat’
  • Removed all rules for user-agent ‘DeepSeekBot’
  • … 16 more change(s)
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/threat-reports/pfpt-us-tr-2026-ai-era-ransomware.pdf for *
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
General Mills generalmills.com modified +9 −2
  • Added Disallow: /*.aspx for *
  • Added Disallow: /*p=1 for *
  • Added Disallow: /.well-known/apple-app-site-association for *
  • Added Disallow: /apple-app-site-association for *
  • Added Disallow: /search for *
  • Added Disallow: /shared/ for *
  • … 4 more change(s)
view diff → snapshot
Barracuda barracuda.com removed +0 −232
  • robots.txt is no longer served (HTTP 404)
Weibo weibo.com modified +35 −15
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘ChatGLM’ entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • Blocked user-agent ‘DeepSeek’ entirely
  • Blocked user-agent ‘DeepSeek Chat’ entirely
  • Blocked user-agent ‘DeepSeekBot’ entirely
  • … 16 more change(s)
view diff → snapshot
Rapid7 rapid7.com modified +0 −2
  • Removed Disallow: /*&page=* for *
  • Removed Disallow: /*?page=* for *
view diff → snapshot
Kasada kasada.io modified +6 −6
  • Added Allow: / for *
  • Added Disallow: /admin for *
  • Removed Allow: /wp-admin/admin-ajax.php for *
  • Removed Disallow: /?s= for *
  • Removed Disallow: /search/ for *
  • Removed Disallow: /wp-admin/ for *
  • … 2 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot