NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
Clorox thecloroxcompany.com modified +7 −7
  • Added Disallow: https://www.thecloroxcompany.com/blog/promoting-community-well-being-with-27-new-grants/ for *
  • Added Disallow: https://www.thecloroxcompany.com/brands/what-were-made-of/smart-label/ingredients-post/ for *
  • Added Disallow: https://www.thecloroxcompany.com/incident/ for *
  • Added Disallow: https://www.thecloroxcompany.com/press-releases/ for *
  • Added Disallow: https://www.thecloroxcompany.com/release/ for *
  • Added Disallow: https://www.thecloroxcompany.com/search/ for *
  • … 8 more change(s)
view diff → snapshot
Labcorp labcorp.com modified +1 −1
  • Added Disallow: /content/dam/labcorp/files/contact-lists/English_HowToContactUs.pdf for *
view diff → snapshot
Air France-KLM airfranceklm.com modified +6 −0
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
view diff → snapshot
Costa Rica (Presidencia) presidencia.go.cr modified +128 −70
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Blocked user-agent ‘atlassian-bot’ entirely
  • Blocked user-agent ‘AwarioRssBot’ entirely
  • Blocked user-agent ‘AwarioSmartBot’ entirely
  • … 45 more change(s)
view diff → snapshot
The Telegraph telegraph.co.uk modified +69 −7
  • Removed Disallow: /promotions/emails/ for *
  • Removed Disallow: /sponsored/travel/msc-cruises/ for *
  • 🤖 AI Added Allow: /bah-avis/usa-sports-road-trip-map for AI2Bot
  • 🤖 AI Added Allow: /bespoke-content/ for AI2Bot
  • 🤖 AI Added Allow: /betting/ for AI2Bot
  • 🤖 AI Added Allow: /brand-usa for AI2Bot
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +27 −0
  • 🤖 AI Added Allow: /blog/how-to-put-your-design-system-to-work-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/figma-deepens-investment-in-india/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/introducing-figma-specs/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/sightlines-issue-no3/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/figma-deepens-investment-in-india/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/introducing-figma-specs/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Snyk snyk.io modified +4 −15
  • Removed Disallow: /*.json$ for *
  • Removed Disallow: /*_buildManifest.js$ for *
  • Removed Disallow: /*_middlewareManifest.js$ for *
  • Removed Disallow: /*_ssgManifest.js$ for *
view diff → snapshot
NBC News nbcnews.com modified +9 −8
  • 🤖 AI Added Allow: /information/nbc-news-info/about-nbc-news-digital-n1232178 for Meta-WebIndexer
  • 🤖 AI Added Allow: /information/nbc-news-info/closed-captioning-n1307063 for Meta-WebIndexer
  • 🤖 AI Added Allow: /information/nbc-news-info/contact-us-n1232521 for Meta-WebIndexer
  • 🤖 AI Added Allow: /news/us-news/nbc-news-digital-editors-n893846 for Meta-WebIndexer
  • 🤖 AI Added Allow: /select for Meta-WebIndexer
  • 🤖 AI Added Allow: /select/ for Meta-WebIndexer
  • … 28 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/lufthansa-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/lufthansa-figma-make/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/lufthansa-figma-make/$ for Claude-User
  • 🤖 AI Added Allow: /blog/lufthansa-figma-make/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/lufthansa-figma-make/$ for PerplexityBot
view diff → snapshot
Surfer surferseo.com modified +1 −5
  • Added Disallow: (empty) for *
  • Removed Disallow: /archive/ for *
  • Removed Disallow: /blog?page= for *
  • Removed Disallow: /design-system/ for *
  • Removed Disallow: /dev/ for *
  • Removed Disallow: /free-outline-generator/?permalink=* for *
view diff → snapshot