NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2614snapshots
1442changes detected
231this week
All changes 🤖 AI-crawler changes Hiding initial archives
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/workflow-lab-deploying-designs-directly-with-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/design-explore/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/workflow-lab-deploying-designs-directly-with-figma-make/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /es-la/design-explore/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/workflow-lab-deploying-designs-directly-with-figma-make/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /es-la/design-explore/$ for OAI-SearchBot
  • … 2 more change(s)
view diff → snapshot
Ford Motor ford.com modified +46 −0
  • Blocked user-agent ‘amazon-kendra’ entirely
  • Blocked user-agent ‘amazon-QBusiness’ entirely
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Blocked user-agent ‘bedrockbot’ entirely
  • … 24 more change(s)
view diff → snapshot
Europe PMC europepmc.org modified +7 −98
  • Removed all rules for user-agent ‘baiduspider’
  • Removed all rules for user-agent ‘bingbot’
  • 🤖 AI Removed rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Removed rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Removed rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Removed rules for AI crawler PerplexityBot (Perplexity)
  • … 21 more change(s)
view diff → snapshot
Le Monde lemonde.fr modified +54 −0
  • Blocked user-agent ‘AIWebIndex’ entirely
  • Blocked user-agent ‘amazon-kendra’ entirely
  • Blocked user-agent ‘amazon-QBusiness’ entirely
  • Blocked user-agent ‘bedrockbot’ entirely
  • Blocked user-agent ‘Crawlspace’ entirely
  • Blocked user-agent ‘Doubaobot’ entirely
  • … 12 more change(s)
view diff → snapshot
Barracuda barracuda.com created +28 −28
  • Added Disallow: /ao4oVNcRKejUUUcGuNvfODmx1iIeG0WoFyjrrlAaARk=.html for *
  • Removed Disallow: /tUBYNifRs1e8unWE00sUsxLRZN11tkvP-DXPflxM6aY=.html for *
  • Added Disallow: /ao4oVNcRKejUUUcGuNvfODmx1iIeG0WoFyjrrlAaARk=.html for DotBot
  • Removed Disallow: /tUBYNifRs1e8unWE00sUsxLRZN11tkvP-DXPflxM6aY=.html for DotBot
  • Added Disallow: /ao4oVNcRKejUUUcGuNvfODmx1iIeG0WoFyjrrlAaARk=.html for sistrix
  • Removed Disallow: /tUBYNifRs1e8unWE00sUsxLRZN11tkvP-DXPflxM6aY=.html for sistrix
  • … 2 more change(s)
view diff → snapshot
Semrush semrush.com modified +0 −4
  • Removed Disallow: /features/for-* for *
  • Removed Disallow: /features/index2/ for *
view diff → snapshot
Barracuda barracuda.com removed +0 −232
  • robots.txt is no longer served (HTTP 404)
Goodyear Tire & Rubber goodyear.com modified +40 −0
  • Blocked user-agent ‘ahrefsbot’ entirely
  • Blocked user-agent ‘cazoodlebot’ entirely
  • Blocked user-agent ‘dotbot/1.0’ entirely
  • Blocked user-agent ‘gigabot’ entirely
  • Blocked user-agent ‘mj12bot’ entirely
  • Blocked user-agent ‘sogou inst spider’ entirely
  • … 20 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Figma Community figma.com modified +10 −0
  • 🤖 AI Added Allow: /blog/figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/how-trackman-built-a-design-first-culture-and-a-competitive-edge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/7-tips-for-using-figma-make-credits-more-efficiently/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/7-tips-for-using-figma-make-credits-more-efficiently/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/7-tips-for-using-figma-make-credits-more-efficiently/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/7-tips-for-using-figma-make-credits-more-efficiently/$ for CHATGPT-User
  • … 34 more change(s)
view diff → snapshot
Goodyear Tire & Rubber goodyear.com modified +0 −40
  • Removed all rules for user-agent ‘ahrefsbot’
  • Removed all rules for user-agent ‘cazoodlebot’
  • Removed all rules for user-agent ‘dotbot/1.0’
  • Removed all rules for user-agent ‘gigabot’
  • Removed all rules for user-agent ‘mj12bot’
  • Removed all rules for user-agent ‘sogou inst spider’
  • … 20 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /customers/mitsubishi-electric-corporation/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/mitsubishi-electric-corporation/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /customers/mitsubishi-electric-corporation/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /customers/mitsubishi-electric-corporation/$ for PerplexityBot
view diff → snapshot