NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2663snapshots
1493changes detected
221this week
All changes 🤖 AI-crawler changes Showing initial archives
Corriere della Sera corriere.it modified +10 −0
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘Diffbot-User’ entirely
  • Blocked user-agent ‘Scrapy’ entirely
  • 🤖 AI Blocked Timpibot (Timpi) entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
view diff → snapshot
DataDome datadome.co modified +37 −0
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Added rules for user-agent ‘Bingbot’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler cohere-ai (Cohere)
  • … 4 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Corriere della Sera corriere.it modified +0 −10
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • Removed all rules for user-agent ‘Diffbot-User’
  • Removed all rules for user-agent ‘Scrapy’
  • 🤖 AI Unblocked Timpibot (Timpi) - rules removed
  • 🤖 AI Unblocked YouBot (You.com) - rules removed
view diff → snapshot
Sanmina sanmina.com initial
  • First snapshot of robots.txt archived
snapshot
Paccar paccar.com initial
  • First snapshot of robots.txt archived
snapshot
Ikea ikea.com initial
  • First snapshot of robots.txt archived
snapshot
Olx olx.com.br initial
  • First snapshot of robots.txt archived
snapshot
Coupang coupang.com initial
  • First snapshot of robots.txt archived
snapshot
Flipkart flipkart.com initial
  • First snapshot of robots.txt archived
snapshot
IGN ign.com initial
  • First snapshot of robots.txt archived
snapshot
Epicgames epicgames.com initial
  • First snapshot of robots.txt archived
snapshot
Yelp yelp.com initial
  • First snapshot of robots.txt archived
snapshot
Discogs discogs.com initial
  • First snapshot of robots.txt archived
snapshot
Kellanova kellanova.com initial
  • First snapshot of robots.txt archived
snapshot
Grok grok.com initial
  • First snapshot of robots.txt archived
snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot