NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Showing initial archives
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
The Trade Desk thetradedesk.com modified +1 −1
  • Added sitemap https://careers.thetradedesk.com/sitemap.xml
  • Removed sitemap https://www.thetradedesk.com/sitemap.xml
view diff → snapshot
Cummins cummins.com modified +1 −1
  • Added Disallow: /*/mp-resource/* for *
  • Removed Disallow: /*/mp-resource/en-na/* for *
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Tata Consultancy Services tcs.com modified +2 −0
  • Added Disallow: /content/dam/global-tcs/en/pdfs/what-we-do/platforms/add/add-analytics-insight-solution.pdf for *
  • Added Disallow: /content/dam/global-tcs/en/pdfs/what-we-do/platforms/tcs-add/solution/tcs-add-analytics-insights-intelligent-decision-making.pdf for *
view diff → snapshot
Renault Group renaultgroup.com modified +0 −72
  • 🤖 AI Unblocked ClaudeBot (Anthropic)
  • 🤖 AI Removed Allow: /br/ for ClaudeBot
  • 🤖 AI Removed Allow: /de/ for ClaudeBot
  • 🤖 AI Removed Allow: /es/ for ClaudeBot
  • 🤖 AI Removed Allow: /it/ for ClaudeBot
  • 🤖 AI Removed Allow: /pt/ for ClaudeBot
  • … 8 more change(s)
view diff → snapshot
DOAJ doaj.org modified +20 −68
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 21 more change(s)
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Indeed indeed.com modified +30 −0
  • Added Allow: /*&start=0& for *
  • Added Allow: /*&start=10& for *
  • Added Allow: /*&start=20& for *
  • Added Allow: /*&start=30& for *
  • Added Allow: /*&start=40& for *
  • Added Allow: /*&start=50& for *
  • … 45 more change(s)
view diff → snapshot
eBay ebay.com modified +5 −1
  • Added Disallow: /d/graphql for *
  • Added Disallow: /looking-glass-proxy/graphql for *
  • Added Disallow: /d/graphql for bingbot
  • Added Disallow: /looking-glass-proxy/graphql for bingbot
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot