NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
Molina Healthcare molinahealthcare.com modified +3 −28
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /marketplace/az/* for *
  • Removed Disallow: /marketplace/ca/* for *
  • Removed Disallow: /marketplace/fl/* for *
  • Removed Disallow: /marketplace/ga/* for *
  • … 24 more change(s)
view diff → snapshot
Heineken heineken.com modified +10 −0
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Google-Extended (Google)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • … 2 more change(s)
view diff → snapshot
Tripadvisor tripadvisor.com modified +3 −18
  • Added Disallow: /AdsManagerAjax for *
  • Added Disallow: /AdsManagerLanderAjax for *
  • Added Disallow: /PlanModeTools for *
  • Removed Disallow: /AdsManager for *
  • Removed Disallow: /GuideAjax for *
  • Removed Disallow: /GuideEditor for *
  • … 45 more change(s)
view diff → snapshot
ElevenLabs elevenlabs.io modified +0 −5
  • Removed Disallow: /sentry-monitoring for *
  • Removed Disallow: /sentry-monitoring for facebookexternalhit
  • Removed Disallow: /sentry-monitoring for LinkedInBot
  • Removed Disallow: /sentry-monitoring for Twitterbot
  • Removed Disallow: /sentry-monitoring for WhatsApp
view diff → snapshot
You.com you.com modified +2 −0
  • Added sitemap https://you.com/sitemaps/docs/sitemap.xml
  • Added sitemap https://you.com/sitemaps/marketing/sitemap.xml
view diff → snapshot
Kohl's kohls.com modified +6 −0
  • Blocked user-agent ‘Y!J-ASR’ entirely
  • Blocked user-agent ‘YandexBot’ entirely
view diff → snapshot
Eastman Chemical eastman.com modified +4 −8
  • Added Disallow: /*product-inquiry for *
  • Added Disallow: /*search-results for *
  • Removed Disallow: */product-inquiry for *
  • Removed Disallow: */search-results for *
view diff → snapshot
Snapchat snapchat.com modified +1 −0
  • Added sitemap https://www.snapchat.com/sitemaps/lens_creators/sitemap-index.xml
view diff → snapshot
Google google.com modified +1 −0
  • Added Disallow: /travel/lodging/imp for *
  • Added Disallow: /travel/lodging/imp for Yandex
view diff → snapshot