NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2673snapshots
1503changes detected
231this week
All changes 🤖 AI-crawler changes Showing initial archives
Peec AI peec.ai initial
  • First snapshot of robots.txt archived
snapshot
W.W. Grainger grainger.com modified +188 −167
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • Blocked user-agent ‘AhrefsBot’ entirely
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • Blocked user-agent ‘Baiduspider’ entirely
  • Blocked user-agent ‘Baiduspider-render’ entirely
  • … 19 more change(s)
view diff →
Regeneron Pharmaceuticals regeneron.com modified +92 −1529
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • … 12 more change(s)
view diff →
Evonik evonik.com modified +47 −22343
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Added rules for user-agent ‘Bingbot’
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • … 25 more change(s)
view diff →
Nikkei nikkei.com modified +143 −10
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
  • Blocked user-agent ‘AI2Bot-Dolma’ entirely
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • … 41 more change(s)
view diff →
Khan Academy khanacademy.org modified +33 −87
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked GPTBot (OpenAI) entirely
  • Added sitemap https://www.khanacademy.org/sitemap.xml
view diff →
American Financial Group afginc.com modified +65 −2036
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • Blocked user-agent ‘facebookcatalog’ entirely
  • Blocked user-agent ‘facebookexternalhit’ entirely
  • 🤖 AI Blocked meta-externalagent (Meta) entirely
view diff →
Etsy etsy.com initial
  • First snapshot of robots.txt archived
snapshot
Wish wish.com initial
  • First snapshot of robots.txt archived
snapshot
AliExpress aliexpress.com initial
  • First snapshot of robots.txt archived
snapshot
Adorama adorama.com initial
  • First snapshot of robots.txt archived
snapshot
Rakuten rakuten.com initial
  • First snapshot of robots.txt archived
snapshot
Shein shein.com initial
  • First snapshot of robots.txt archived
snapshot
Temu temu.com initial
  • First snapshot of robots.txt archived
snapshot
JD.com jd.com initial
  • First snapshot of robots.txt archived
snapshot
HashiCorp hashicorp.com initial
  • First snapshot of robots.txt archived
snapshot
Coveo coveo.com initial
  • First snapshot of robots.txt archived
snapshot
Mediahuis mediahuis.com initial
  • First snapshot of robots.txt archived
snapshot
DataDome datadome.co initial
  • First snapshot of robots.txt archived
snapshot