NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
All changes 🤖 AI-crawler changes Hiding initial archives
ElevenLabs elevenlabs.io modified +4 −0
  • Added Disallow: /nl$ for *
  • Added Disallow: /nl/ for *
  • Added Disallow: /nl$ for Twitterbot
  • Added Disallow: /nl/ for Twitterbot
view diff → snapshot
NRG Energy nrg.com modified +5 −11
  • Added Disallow: */search for *
  • Added Disallow: *?keyword=* for *
  • Removed Disallow: */campaigns for *
  • Removed Disallow: /*.html/plans/ for *
  • Removed Disallow: /business/*.html/plans/ for *
  • Removed Disallow: /contact-us/thank-you.html for *
  • … 4 more change(s)
view diff → snapshot
IGN ign.com modified +86 −3
  • Blocked user-agent ‘AgentDataBot’ entirely
  • Blocked user-agent ‘AIWebIndex’ entirely
  • Blocked user-agent ‘AIWebIndex-Agent’ entirely
  • Blocked user-agent ‘amazon-kendra’ entirely
  • Blocked user-agent ‘amazon-QBusiness’ entirely
  • Blocked user-agent ‘Amzn-User’ entirely
  • … 45 more change(s)
view diff → snapshot
Simon & Schuster simonandschuster.com modified +8 −1
  • Added Allow: /search/books/Available-For-Sale-Now/_/N-pgz/Ne-pgx$ for *
  • Added Allow: /search/books/Category-Biography-Autobiography/Available-For-Sale-Now/_/N-hr0Zpgz/Ne-pgx$ for *
  • Added Allow: /search/books/Category-Fiction-Mystery-Detective/Available-For-Sale-Now/_/N-g32Zpgz/Ne-pgx$ for *
  • Added Allow: /search/books/Category-Fiction/Available-For-Sale-Now/_/N-g1hZpgz/Ne-pgx$ for *
  • Added Allow: /search/books/Category-Games-Crosswords/Available-For-Sale-Now/_/N-g6aZpgz/Ne-ffk$ for *
  • Added Allow: /search/books/Category-History/Available-For-Sale-Now/_/N-ha4Zpgz/Ne-ffk$ for *
  • … 3 more change(s)
view diff → snapshot