NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Showing initial archives
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/3d-scene-generator-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-brand-video-generator/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +3 −0
  • 🤖 AI Added Allow: /es-la/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/get-started-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/how-we-secure-figmas-internal-systems-with-agents/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /pt-br/blog/how-we-secure-figmas-internal-systems-with-agents/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /webinars/get-started-with-the-figma-agent/$ for Claude-SearchBot
  • … 9 more change(s)
view diff → snapshot
Ahrefs ahrefs.com modified +0 −8
  • Removed Allow: /de/blog for *
  • Removed Allow: /es/blog for *
  • Removed Allow: /fr/blog for *
  • Removed Allow: /it/blog for *
  • Removed Allow: /ja/blog for *
  • Removed Allow: /pt/blog for *
  • … 2 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Nvidia nvidia.com modified +18 −69
  • 🤖 AI Removed rules for AI crawler AI2Bot (Allen Institute for AI)
  • Removed all rules for user-agent ‘Amazon-kendra’
  • 🤖 AI Removed rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Removed rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Removed rules for AI crawler Applebot-Extended (Apple)
  • Removed all rules for user-agent ‘AutoGPT’
  • … 45 more change(s)
view diff → snapshot
Ecolab ecolab.com modified +29 −1
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
  • Added sitemap https://ecolab-dotcom-prod.vercel.app/sitemap-llm.xml
  • … 2 more change(s)
view diff → snapshot