NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
EPFL epfl.ch modified +130 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Blocked user-agent ‘atlassian-bot’ entirely
  • Blocked user-agent ‘AwarioRssBot’ entirely
  • … 27 more change(s)
view diff → snapshot
DBLP dblp.org modified +66 −165
  • Removed all rules for user-agent ‘AddSearchBot’
  • Removed all rules for user-agent ‘AgentTimes’
  • 🤖 AI Unblocked AI2Bot (Allen Institute for AI) - rules removed
  • Removed all rules for user-agent ‘AI2Bot-DeepResearchEval’
  • Removed all rules for user-agent ‘Ai2Bot-Dolma’
  • Removed all rules for user-agent ‘aiHitBot’
  • … 45 more change(s)
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Walmart walmart.com modified +0 −3
  • Removed sitemap https://www.walmart.com/sitemap_slp_01.xml
  • Removed sitemap https://www.walmart.com/sitemap_slp_02.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Arrow Electronics arrow.com removed +0 −116
  • robots.txt is no longer served (HTTP 404)
Copart copart.com modified +39 −1
  • Added rules for user-agent ‘*’
  • Added sitemap https://www.copart-sitemaps.com/ar/sitemap-index.xml
  • Added sitemap https://www.copart-sitemaps.com/es/sitemap-index.xml
  • Added sitemap https://www.copart-sitemaps.com/fr-CA/sitemap-index.xml
  • Added sitemap https://www.copart-sitemaps.com/pl/sitemap-index.xml
  • Added sitemap https://www.copart-sitemaps.com/ru/sitemap-index.xml
  • … 1 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot