NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2658snapshots
1487changes detected
242this week
All changes 🤖 AI-crawler changes Hiding initial archives
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
W.W. Grainger grainger.com modified +188 −167
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • Blocked user-agent ‘AhrefsBot’ entirely
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • Blocked user-agent ‘Baiduspider’ entirely
  • Blocked user-agent ‘Baiduspider-render’ entirely
  • … 19 more change(s)
view diff →
Regeneron Pharmaceuticals regeneron.com modified +92 −1529
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • … 12 more change(s)
view diff →
Evonik evonik.com modified +47 −22343
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Added rules for user-agent ‘Bingbot’
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • … 25 more change(s)
view diff →
Nikkei nikkei.com modified +143 −10
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
  • Blocked user-agent ‘AI2Bot-Dolma’ entirely
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • … 41 more change(s)
view diff →
Khan Academy khanacademy.org modified +33 −87
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked GPTBot (OpenAI) entirely
  • Added sitemap https://www.khanacademy.org/sitemap.xml
view diff →
American Financial Group afginc.com modified +65 −2036
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • Blocked user-agent ‘facebookcatalog’ entirely
  • Blocked user-agent ‘facebookexternalhit’ entirely
  • 🤖 AI Blocked meta-externalagent (Meta) entirely
view diff →
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
F5 f5.com modified +10 −20
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • Added sitemap https://www.f5.com/labs/sitemap.xml
  • Added sitemap https://www.f5.com/sitemap.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
F5 f5.com modified +1 −1
  • Formatting/comment-only change (no semantic difference)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot