NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2667snapshots
1497changes detected
225this week
All changes 🤖 AI-crawler changes Showing initial archives
Corriere della Sera corriere.it modified +10 −0
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘Diffbot-User’ entirely
  • Blocked user-agent ‘Scrapy’ entirely
  • 🤖 AI Blocked Timpibot (Timpi) entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
view diff → snapshot
Perplexity perplexity.ai modified +2 −0
  • Added Disallow: /marketing/prerelease/ for *
  • Added Disallow: /marketing/prerelease/ for Googlebot
view diff → snapshot
SoundCloud soundcloud.com modified +60 −29
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Blocked cohere-ai (Cohere) entirely
  • … 45 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Barracuda barracuda.com created +28 −28
  • Added Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for *
  • Removed Disallow: /H1Rj9-8bvN3I9mQlIPVT5Cm31oOG5HamhkmPLNbEbyE=.html for *
  • Added Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for DotBot
  • Removed Disallow: /H1Rj9-8bvN3I9mQlIPVT5Cm31oOG5HamhkmPLNbEbyE=.html for DotBot
  • Added Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for sistrix
  • Removed Disallow: /H1Rj9-8bvN3I9mQlIPVT5Cm31oOG5HamhkmPLNbEbyE=.html for sistrix
  • … 2 more change(s)
view diff → snapshot
BuzzFeed buzzfeed.com modified +1 −0
  • 🤖 AI Added Allow: /rss-feeds/feeds/bf-amazon-ai-grounding.extended.xml for Amazonbot
view diff → snapshot
Cencora cencora.com modified +3 −1
  • Added Disallow: /Global-reach/Europe/United-Kingdom/Legal-regulatory-hub/Almus-NetGen-Terms-Conditions for *
  • Added Disallow: /Global-reach/Europe/United-Kingdom/Legal-regulatory-hub/BE-Force-terms-conditions for *
  • Added Disallow: /Global-reach/Europe/United-Kingdom/Legal-regulatory-hub/pharmacyone-brand-equalisation-terms for *
  • Removed Disallow: /web-team-requests for *
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Bild bild.de modified +4 −2
  • Added Disallow: /cmsid/ for *
  • Added Disallow: /sonstiges/vorproduktion/ for *
  • Removed Disallow: /vorprod/ for *
  • Added Disallow: /cmsid/ for Googlebot-News
  • Added Disallow: /sonstiges/vorproduktion/ for Googlebot-News
  • Removed Disallow: /vorprod/ for Googlebot-News
view diff → snapshot
Figma Community figma.com modified +94 −0
  • 🤖 AI Added Allow: /de-de/resource-library/ki-coding-tools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/resource-library/was-ist-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/resource-library/herramientas-programacion-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/resource-library/que-es-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/2025-shifting-roles-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/4-ways-for-design-teams-to-chart-new-territory-with-figma-make/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Barracuda barracuda.com removed +0 −232
  • robots.txt is no longer served (HTTP 404)
Cencora cencora.com modified +0 −32
  • Removed Disallow: /initiatives/pharma/formularydecisions-connection-newsletter/* for *
  • Removed Disallow: /initiatives/pharma/formularydecisions-insider-newsletter/* for *
view diff → snapshot
Luxembourg (Gouvernement) gouvernement.lu modified +7 −0
  • Added Allow: /*?b=* for *
  • Added Disallow: /*?* for *
  • Added Disallow: /*?*&b=* for *
  • Added Disallow: /*?*&q=* for *
  • Added Disallow: /*?*&r=* for *
  • Added Disallow: /*?b=*&* for *
  • … 1 more change(s)
view diff → snapshot
AutoZone autozone.com modified +0 −5
  • Removed Disallow: /sls/pd/product-navigation-search/v1/solr-search/autozone_diy_usa_concepts_live/semanticSearch* for *
  • Removed Disallow: /sls/pd/product-navigation-search/v1/solr-search/autozone_ims_concepts_live-mex/semanticSearch* for *
  • Removed sitemap https://www.autozone.com/locations/sitemap.xml
view diff → snapshot