NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2625snapshots
1453changes detected
242this week
All changes 🤖 AI-crawler changes Hiding initial archives
ChatGPT chatgpt.com modified +6 −0
  • Added Allow: /*/import-to-chatgpt$ for *
  • Added Allow: /*/remote$ for *
  • Added Allow: /*/work$ for *
  • Added Allow: /import-to-chatgpt$ for *
  • Added Allow: /remote$ for *
  • Added Allow: /work$ for *
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Take-Two Interactive take2games.com removed +0 −9
  • robots.txt is no longer served (HTTP 404)
Figma Community figma.com modified +3 −1
  • 🤖 AI Added Allow: /blog/gpt-5-6-is-now-available-in-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/jen-dunnam-lead-design-teams/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/sightlines/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /blog/jen-dunham-lead-design-teams/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/gpt-5-6-is-now-available-in-figma-make/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/jen-dunnam-lead-design-teams/$ for Claude-SearchBot
  • … 10 more change(s)
view diff → snapshot
Anysphere (Cursor) cursor.com modified +11 −0
  • Added Allow: /docs-static/apple-touch-icon* for *
  • Added Allow: /docs-static/favicon* for *
  • Added Allow: /docs-static/icon-* for *
  • Added Allow: /marketing-static/apple-touch-icon* for *
  • Added Allow: /marketing-static/favicon* for *
  • Added Allow: /marketing-static/icon-* for *
view diff → snapshot
Roblox roblox.com modified +28 −0
  • Added sitemap https://www.roblox.com/sitemap-ar.xml
  • Added sitemap https://www.roblox.com/sitemap-de.xml
  • Added sitemap https://www.roblox.com/sitemap-es.xml
  • Added sitemap https://www.roblox.com/sitemap-fr.xml
  • Added sitemap https://www.roblox.com/sitemap-games-ar.xml
  • Added sitemap https://www.roblox.com/sitemap-games-de.xml
  • … 22 more change(s)
view diff → snapshot
Volvo Cars volvocars.com modified +7 −8
  • Removed all rules for user-agent ‘Googlebot’
  • Added Allow: /*/build/$ for *
  • Added Allow: /build/$ for *
  • Added Disallow: /*/build/ for *
  • Added Disallow: /build/ for *
  • Added Allow: /zh-cn/ for Baiduspider
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +4 −0
  • 🤖 AI Added Allow: /blog/jen-dunham-lead-design-teams/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/botim-ai-ready-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/customers/botim-ai-ready-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/claude-skills-for-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/jen-dunham-lead-design-teams/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /customers/botim-ai-ready-design/$ for Claude-SearchBot
  • … 10 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Barracuda barracuda.com created +28 −28
  • Added Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for *
  • Removed Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for *
  • Added Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for DotBot
  • Removed Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for DotBot
  • Added Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for sistrix
  • Removed Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for sistrix
  • … 2 more change(s)
view diff → snapshot
Barracuda barracuda.com removed +0 −232
  • robots.txt is no longer served (HTTP 404)
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/e-books/pfpt-us-eb-email-security-blind-spots.pdf for *
view diff → snapshot
X (Twitter) x.com modified +3 −3
  • Added Disallow: /search?q= for Bingbot
  • Removed Allow: /search?q=%23 for Bingbot
  • Added Disallow: /search?q= for facebookexternalhit
  • Removed Allow: /search?q=%23 for facebookexternalhit
  • Added Disallow: /search?q= for Googlebot
  • Removed Allow: /search?q=%23 for Googlebot
view diff → snapshot