NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2643snapshots
1471changes detected
226this week
All changes 🤖 AI-crawler changes Hiding initial archives
Ericsson ericsson.com removed +0 −27
  • robots.txt is no longer served (HTTP 404)
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for PerplexityBot
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
IMDb imdb.com modified +0 −32
  • Removed all rules for user-agent ‘360Spider’
  • Removed all rules for user-agent ‘AhrefsBot’
  • Removed all rules for user-agent ‘archive.org_bot’
  • Removed all rules for user-agent ‘ArchiveBot’
  • Removed all rules for user-agent ‘BingPreview’
  • Removed all rules for user-agent ‘BrightEdge’
  • … 26 more change(s)
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/building-accessibility-into-a-canvas-based-product/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/building-accessibility-into-a-canvas-based-product/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/building-accessibility-into-a-canvas-based-product/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for OAI-SearchBot
  • … 2 more change(s)
view diff → snapshot
Microsoft microsoft.com modified +9 −1
  • Added Allow: /en/microsoft-365/free-office-online-for-the-web for *
  • Added Allow: /en/microsoft-365/onenote/digital-note-taking-app for *
  • Added Allow: /en/microsoft-365/outlook/web-email-login-for-outlook for *
  • Added Allow: /en/microsoft-365/planner/microsoft-planner for *
  • Added Allow: /en/microsoft-365/products-apps-services for *
  • Added Allow: /en/microsoft-365/publisher for *
  • … 2 more change(s)
view diff → snapshot
Analog Devices analog.com modified +4 −0
  • Added Disallow: /cn/crsearch/ for *
  • Added Disallow: /en/crsearch/ for *
  • Added Disallow: /jp/crsearch/ for *
  • Added Disallow: /sitecore/ for *
view diff → snapshot