NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Showing initial archives
Molina Healthcare molinahealthcare.com modified +28 −3
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /marketplace/az/* for *
  • Added Disallow: /marketplace/ca/* for *
  • Added Disallow: /marketplace/fl/* for *
  • Added Disallow: /marketplace/ga/* for *
  • Added Disallow: /marketplace/ia/* for *
  • … 24 more change(s)
view diff → snapshot
Realtor realtor.com modified +1 −5
  • Removed Allow: /realestateandhomes-search/*price-na- for *
  • Removed Disallow: /realestateandhomes-search/*%2C for *
  • Removed Disallow: /realestateandhomes-search/*%2c for *
  • Removed Disallow: /realestateandhomes-search/*, for *
  • Removed Disallow: /realestateandhomes-search/*/price- for *
view diff → snapshot
Tripadvisor tripadvisor.com modified +3 −32
  • Added Disallow: /MobileLocationPhotoDirectLink for *
  • Added Disallow: /MobileLocationPhotos for *
  • Added Disallow: /TripConnectPaymentFormV2 for *
  • Removed Disallow: /AffiliateWidgets for *
  • Removed Disallow: /DestinationArticle for *
  • Removed Disallow: /Destinations for *
  • … 45 more change(s)
view diff → snapshot
Cisco Systems cisco.com modified +0 −19
  • Removed all rules for user-agent ‘Bingbot’
  • 🤖 AI Removed rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Removed rules for AI crawler ClaudeBot (Anthropic)
  • Removed all rules for user-agent ‘Google AdsBot’
  • 🤖 AI Removed rules for AI crawler Google-Extended (Google)
  • Removed all rules for user-agent ‘Googlebot’
  • … 4 more change(s)
view diff → snapshot
Sysco sysco.com modified +0 −1
  • Removed Disallow: /sysco-distribution-food-safety-and-quality-assurance for *
view diff → snapshot
Schneider Electric se.com modified +3 −0
  • Added sitemap https://www.se.com/ca/en/home/sitemap.xml
  • Added sitemap https://www.se.com/ca/fr/home/sitemap.xml
  • Added sitemap https://www.se.com/us/en/home/sitemap.xml
view diff → snapshot
Figma Community figma.com modified +20 −0
  • 🤖 AI Added Allow: /es-es/agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /hi-in/agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /id-id/agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/agent/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +3 −28
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /marketplace/az/* for *
  • Removed Disallow: /marketplace/ca/* for *
  • Removed Disallow: /marketplace/fl/* for *
  • Removed Disallow: /marketplace/ga/* for *
  • … 24 more change(s)
view diff → snapshot
EPFL epfl.ch modified +0 −130
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked anthropic-ai (Anthropic) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • Removed all rules for user-agent ‘atlassian-bot’
  • Removed all rules for user-agent ‘AwarioRssBot’
  • … 27 more change(s)
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Alamy alamy.com modified +3 −0
  • Added Disallow: /fpanalytics/ for *
  • Added Disallow: /purchased-assets/ for *
  • Added Disallow: /search-assistant/ for *
  • Added Disallow: /fpanalytics/ for AdsBot-Google
  • Added Disallow: /purchased-assets/ for AdsBot-Google
  • Added Disallow: /search-assistant/ for AdsBot-Google
  • … 45 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +29 −2
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /marketplace/az/* for *
  • Added Disallow: /marketplace/ca/* for *
  • Added Disallow: /marketplace/fl/* for *
  • Added Disallow: /marketplace/ga/* for *
  • Added Disallow: /marketplace/ia/* for *
  • … 23 more change(s)
view diff → snapshot