NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2658snapshots
1487changes detected
242this week
All changes 🤖 AI-crawler changes Showing initial archives
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
IMDb imdb.com modified +0 −32
  • Removed all rules for user-agent ‘360Spider’
  • Removed all rules for user-agent ‘AhrefsBot’
  • Removed all rules for user-agent ‘archive.org_bot’
  • Removed all rules for user-agent ‘ArchiveBot’
  • Removed all rules for user-agent ‘BingPreview’
  • Removed all rules for user-agent ‘BrightEdge’
  • … 26 more change(s)
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/building-accessibility-into-a-canvas-based-product/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/building-accessibility-into-a-canvas-based-product/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/building-accessibility-into-a-canvas-based-product/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for OAI-SearchBot
  • … 2 more change(s)
view diff → snapshot
Microsoft microsoft.com modified +9 −1
  • Added Allow: /en/microsoft-365/free-office-online-for-the-web for *
  • Added Allow: /en/microsoft-365/onenote/digital-note-taking-app for *
  • Added Allow: /en/microsoft-365/outlook/web-email-login-for-outlook for *
  • Added Allow: /en/microsoft-365/planner/microsoft-planner for *
  • Added Allow: /en/microsoft-365/products-apps-services for *
  • Added Allow: /en/microsoft-365/publisher for *
  • … 2 more change(s)
view diff → snapshot
Analog Devices analog.com modified +4 −0
  • Added Disallow: /cn/crsearch/ for *
  • Added Disallow: /en/crsearch/ for *
  • Added Disallow: /jp/crsearch/ for *
  • Added Disallow: /sitecore/ for *
view diff → snapshot
Volvo Cars volvocars.com modified +5 −2
  • Added rules for user-agent ‘YisouSpider’
  • 🤖 AI Added Disallow: /*/build/* for Amazonbot
  • 🤖 AI Added Disallow: /*/order-request/* for Amazonbot
  • 🤖 AI Removed Disallow: /*/build/*?* for Amazonbot
  • 🤖 AI Removed Disallow: /*/order* for Amazonbot
  • 🤖 AI Added Disallow: /*/build/* for anthropic-ai
  • … 45 more change(s)
view diff → snapshot