NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Hiding initial archives
Figma Community figma.com modified +3 −0
  • 🤖 AI Added Allow: /es-la/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/get-started-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/how-we-secure-figmas-internal-systems-with-agents/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /pt-br/blog/how-we-secure-figmas-internal-systems-with-agents/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /webinars/get-started-with-the-figma-agent/$ for Claude-SearchBot
  • … 9 more change(s)
view diff → snapshot
Ahrefs ahrefs.com modified +0 −8
  • Removed Allow: /de/blog for *
  • Removed Allow: /es/blog for *
  • Removed Allow: /fr/blog for *
  • Removed Allow: /it/blog for *
  • Removed Allow: /ja/blog for *
  • Removed Allow: /pt/blog for *
  • … 2 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Nvidia nvidia.com modified +18 −69
  • 🤖 AI Removed rules for AI crawler AI2Bot (Allen Institute for AI)
  • Removed all rules for user-agent ‘Amazon-kendra’
  • 🤖 AI Removed rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Removed rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Removed rules for AI crawler Applebot-Extended (Apple)
  • Removed all rules for user-agent ‘AutoGPT’
  • … 45 more change(s)
view diff → snapshot
Ecolab ecolab.com modified +29 −1
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
  • Added sitemap https://ecolab-dotcom-prod.vercel.app/sitemap-llm.xml
  • … 2 more change(s)
view diff → snapshot
Figma Community figma.com modified +30 −0
  • 🤖 AI Added Allow: /blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/code-craft-and-the-making-of-nested-folders/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/designer-framework-for-better-ai-prompts/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/how-figma-stays-ahead-of-vulnerabilities-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/code-craft-and-the-making-of-nested-folders/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/designer-framework-for-better-ai-prompts/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
USA Today usatoday.com modified +6 −0
  • Added Allow: /story/shopping/ for ChatGPT Operator
  • 🤖 AI Added Allow: /story/shopping/ for ChatGPT-User
  • 🤖 AI Added Allow: /story/shopping/ for Google-Extended
  • 🤖 AI Added Allow: /story/shopping/ for GPTBot
  • 🤖 AI Added Allow: /story/shopping/ for OAI-SearchBot
  • 🤖 AI Added Allow: /story/shopping/ for PerplexityBot
view diff → snapshot
TotalEnergies totalenergies.com modified +1 −1
  • Added sitemap https://prd-backoffice.totalenergies.com/sitemap.xml
  • Removed sitemap https://totalenergies.com/sitemap.xml
view diff → snapshot
The Trade Desk thetradedesk.com modified +1 −1
  • Added sitemap https://www.thetradedesk.com/sitemap.xml
  • Removed sitemap https://www.thetradedesk.cn/sitemap.xml
view diff → snapshot