NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2653snapshots
1482changes detected
237this week
All changes 🤖 AI-crawler changes Hiding initial archives
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Bild bild.de modified +4 −2
  • Added Disallow: /cmsid/ for *
  • Added Disallow: /sonstiges/vorproduktion/ for *
  • Removed Disallow: /vorprod/ for *
  • Added Disallow: /cmsid/ for Googlebot-News
  • Added Disallow: /sonstiges/vorproduktion/ for Googlebot-News
  • Removed Disallow: /vorprod/ for Googlebot-News
view diff → snapshot
Figma Community figma.com modified +94 −0
  • 🤖 AI Added Allow: /de-de/resource-library/ki-coding-tools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/resource-library/was-ist-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/resource-library/herramientas-programacion-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/resource-library/que-es-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/2025-shifting-roles-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/4-ways-for-design-teams-to-chart-new-territory-with-figma-make/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Barracuda barracuda.com removed +0 −232
  • robots.txt is no longer served (HTTP 404)
Cencora cencora.com modified +0 −32
  • Removed Disallow: /initiatives/pharma/formularydecisions-connection-newsletter/* for *
  • Removed Disallow: /initiatives/pharma/formularydecisions-insider-newsletter/* for *
view diff → snapshot
Luxembourg (Gouvernement) gouvernement.lu modified +7 −0
  • Added Allow: /*?b=* for *
  • Added Disallow: /*?* for *
  • Added Disallow: /*?*&b=* for *
  • Added Disallow: /*?*&q=* for *
  • Added Disallow: /*?*&r=* for *
  • Added Disallow: /*?b=*&* for *
  • … 1 more change(s)
view diff → snapshot
AutoZone autozone.com modified +0 −5
  • Removed Disallow: /sls/pd/product-navigation-search/v1/solr-search/autozone_diy_usa_concepts_live/semanticSearch* for *
  • Removed Disallow: /sls/pd/product-navigation-search/v1/solr-search/autozone_ims_concepts_live-mex/semanticSearch* for *
  • Removed sitemap https://www.autozone.com/locations/sitemap.xml
view diff → snapshot
GE HealthCare gehealthcare.com modified +10 −4
  • Added sitemap https://www.gehealthcare.com/cn.sitemap.xml
  • Added sitemap https://www.gehealthcare.com/global/de.sitemap.xml
  • Added sitemap https://www.gehealthcare.com/global/es.sitemap.xml
  • Added sitemap https://www.gehealthcare.com/global/fr.sitemap.xml
  • Added sitemap https://www.gehealthcare.com/jp.sitemap.xml
  • Added sitemap https://www.gehealthcare.com/kr.sitemap.xml
  • … 8 more change(s)
view diff → snapshot
Perplexity perplexity.ai modified +0 −34
  • Removed Allow: /academic for *
  • Removed Allow: /api-platform for *
  • Removed Allow: /assistant for *
  • Removed Allow: /comet for *
  • Removed Allow: /comet/gettingstarted for *
  • Removed Allow: /comet/resources for *
  • … 28 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
El País elpais.com modified +14 −0
  • 🤖 AI Blocked Claude-SearchBot (Anthropic) entirely
  • Blocked user-agent ‘Comet’ entirely
  • 🤖 AI Blocked Perplexity-User (Perplexity) entirely
  • 🤖 AI Blocked PerplexityBot (Perplexity) entirely
view diff → snapshot
Figma Community figma.com modified +104 −56
  • 🤖 AI Added Allow: /blog/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/agent-custom-tools-context-skills/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/code-on-the-figma-canvas/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/config-2026-recap/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/future-states/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/white-papers/pfpt-jp-wp-securing-and-governing-AI-in-the-modern-enterprise-guide.pdf for *
view diff → snapshot