NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2575snapshots
1402changes detected
229this week
All changes 🤖 AI-crawler changes Hiding initial archives
Figma Community figma.com modified +104 −56
  • 🤖 AI Added Allow: /blog/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/agent-custom-tools-context-skills/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/code-on-the-figma-canvas/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/config-2026-recap/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/future-states/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Ireland (Gov.ie) gov.ie modified +53 −2
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • Blocked user-agent ‘Applebot’ entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Blocked user-agent ‘Baiduspider’ entirely
  • Blocked user-agent ‘Baiduspider-image’ entirely
  • Blocked user-agent ‘Baiduspider-video’ entirely
  • … 13 more change(s)
view diff → snapshot
W.W. Grainger grainger.com modified +188 −167
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • Blocked user-agent ‘AhrefsBot’ entirely
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • Blocked user-agent ‘Baiduspider’ entirely
  • Blocked user-agent ‘Baiduspider-render’ entirely
  • … 19 more change(s)
view diff →
Regeneron Pharmaceuticals regeneron.com modified +92 −1529
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • … 12 more change(s)
view diff →
Evonik evonik.com modified +47 −22343
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Added rules for user-agent ‘Bingbot’
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • … 25 more change(s)
view diff →
Nikkei nikkei.com modified +143 −10
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
  • Blocked user-agent ‘AI2Bot-Dolma’ entirely
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • … 41 more change(s)
view diff →
Khan Academy khanacademy.org modified +33 −87
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked GPTBot (OpenAI) entirely
  • Added sitemap https://www.khanacademy.org/sitemap.xml
view diff →
American Financial Group afginc.com modified +65 −2036
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • Blocked user-agent ‘facebookcatalog’ entirely
  • Blocked user-agent ‘facebookexternalhit’ entirely
  • 🤖 AI Blocked meta-externalagent (Meta) entirely
view diff →