NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
Nikkei nikkei.com modified +143 −10
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
  • Blocked user-agent ‘AI2Bot-Dolma’ entirely
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • … 41 more change(s)
view diff →
Khan Academy khanacademy.org modified +33 −87
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked GPTBot (OpenAI) entirely
  • Added sitemap https://www.khanacademy.org/sitemap.xml
view diff →
American Financial Group afginc.com modified +65 −2036
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • Blocked user-agent ‘facebookcatalog’ entirely
  • Blocked user-agent ‘facebookexternalhit’ entirely
  • 🤖 AI Blocked meta-externalagent (Meta) entirely
view diff →