NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2647snapshots
1476changes detected
231this week
All changes 🤖 AI-crawler changes Hiding initial archives
Corriere della Sera corriere.it modified +10 −0
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘Diffbot-User’ entirely
  • Blocked user-agent ‘Scrapy’ entirely
  • 🤖 AI Blocked Timpibot (Timpi) entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
view diff → snapshot
Corriere della Sera corriere.it modified +0 −10
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • Removed all rules for user-agent ‘Diffbot-User’
  • Removed all rules for user-agent ‘Scrapy’
  • 🤖 AI Unblocked Timpibot (Timpi) - rules removed
  • 🤖 AI Unblocked YouBot (You.com) - rules removed
view diff → snapshot
Unum Group unum.com modified +8 −1
  • Added Allow: / for *
  • Added sitemap https://www.unum.com/sitemap.xml
  • Removed sitemap https://unum.com/sitemap_unum.xml
view diff → snapshot
Corriere della Sera corriere.it modified +10 −0
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘Diffbot-User’ entirely
  • Blocked user-agent ‘Scrapy’ entirely
  • 🤖 AI Blocked Timpibot (Timpi) entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Corriere della Sera corriere.it modified +0 −10
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • Removed all rules for user-agent ‘Diffbot-User’
  • Removed all rules for user-agent ‘Scrapy’
  • 🤖 AI Unblocked Timpibot (Timpi) - rules removed
  • 🤖 AI Unblocked YouBot (You.com) - rules removed
view diff → snapshot