NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
All changes 🤖 AI-crawler changes Hiding initial archives
Figma Community figma.com modified +23 −0
  • 🤖 AI Added Allow: /de-de/blog/source-material-zach-lieberman/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/try-these-5-weave-tools-and-share-your-own/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-dashboard-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-zach-lieberman/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/try-these-5-weave-tools-and-share-your-own/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/agente-de-paneles-de-ia/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Orange orange.com modified +2 −2
  • Removed Disallow: /en/research* for *
  • Removed Disallow: /fr/research* for *
view diff → snapshot
NPR npr.org modified +6 −0
  • 🤖 AI Blocked Claude-SearchBot (Anthropic) entirely
  • 🤖 AI Blocked Claude-User (Anthropic) entirely
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Shell shell.com modified +2 −0
  • Added Disallow: /business-customers/marine/about-shell-marine.html for *
  • Added Disallow: /business-customers/marine/about-shell-marine/ for *
view diff → snapshot
Bank Central Asia bca.co.id modified +9 −0
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
view diff → snapshot
Bayer bayer.com modified +11 −116
  • Added Disallow: /de-de/investoren/document-download-tool? for *
  • Added Disallow: /uk-ua/v-med-news/1 for *
  • Added Disallow: /uk-ua/v-med-news/2 for *
  • Added Disallow: /uk-ua/v-med-news/3 for *
  • Added Disallow: /uk-ua/v-med-news/anticoagulant-therapy for *
  • Added Disallow: /uk-ua/v-med-news/contraception for *
  • … 45 more change(s)
view diff → snapshot
Wiz wiz.io modified +2 −2
  • Added sitemap https://www.wiz.io/sitemap.xml
  • Removed sitemap https://wiz.io/sitemap.xml
view diff → snapshot
Insight Enterprises insight.com modified +91 −19
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • 🤖 AI Blocked GPTBot (OpenAI) entirely
  • 🤖 AI Blocked meta-externalagent (Meta) entirely
  • … 37 more change(s)
view diff → snapshot
TIAA tiaa.org modified +1 −0
  • Added Allow: /public/land/personalized-financial-advice-1 for *
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot