NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5416snapshots
4280changes detected
237this week
All changes 🤖 AI-crawler changes Hiding initial archives
Molina Healthcare molinahealthcare.com modified +3 −28
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /marketplace/az/* for *
  • Removed Disallow: /marketplace/ca/* for *
  • Removed Disallow: /marketplace/fl/* for *
  • Removed Disallow: /marketplace/ga/* for *
  • … 24 more change(s)
view diff → snapshot
Bild bild.de modified +2 −12
  • Added Disallow: /esim/go/ for *
  • Removed Disallow: /energieloesungen/*-form/ for *
  • Removed Disallow: /energieloesungen/social/*-form/ for *
  • Removed Disallow: /energieloesungen/social/go/ for *
  • Removed Disallow: /energieloesungen/vergleich/*-form/ for *
  • Removed Disallow: /energieloesungen/vergleich/go/ for *
  • … 8 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +29 −2
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /marketplace/az/* for *
  • Added Disallow: /marketplace/ca/* for *
  • Added Disallow: /marketplace/fl/* for *
  • Added Disallow: /marketplace/ga/* for *
  • Added Disallow: /marketplace/ia/* for *
  • … 23 more change(s)
view diff → snapshot
Weather weather.com modified +0 −4
  • Removed Disallow: /*/hourbyhour/day- for *
  • Removed Disallow: /*/localalerts?*&phenomena= for *
  • Removed Disallow: /*/localalerts?phenomena= for *
view diff → snapshot
Cigna Group thecignagroup.com modified +77 −1
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.css$ for *
  • Added Allow: /core/*.css? for *
  • Added Allow: /core/*.gif for *
  • Added Allow: /core/*.jpeg for *
  • Added Allow: /core/*.jpg for *
  • … 45 more change(s)
view diff → snapshot
Waste Management wm.com modified +5 −7
  • Removed all rules for user-agent ‘AdsBot-Google’
  • Added Allow: / for *
  • Added Disallow: /api/ for *
  • Added Disallow: /ui-elements for *
  • Added sitemap https://www.wm.com/sitemap.xml
  • Removed sitemap https://www.wm.com/images-sitemap.xml
  • … 2 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +2 −29
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /marketplace/az/* for *
  • Removed Disallow: /marketplace/ca/* for *
  • Removed Disallow: /marketplace/fl/* for *
  • Removed Disallow: /marketplace/ga/* for *
  • … 23 more change(s)
view diff → snapshot
Figma Community figma.com modified +11 −0
  • 🤖 AI Added Allow: /de-de/legal/privacy/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/legal/tos/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/legal/privacy/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/legal/tos/$ for CHATGPT-User
  • 🤖 AI Added Allow: /hi-in/legal/$ for CHATGPT-User
  • 🤖 AI Added Allow: /id-id/legal/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
TotalEnergies totalenergies.com modified +1 −1
  • Added sitemap https://totalenergies.com/sitemap.xml
  • Removed sitemap https://prd-backoffice.totalenergies.com/sitemap.xml
view diff → snapshot
Delta Air Lines delta.com modified +0 −58
  • Removed all rules for user-agent ‘ai2-crawler’
  • 🤖 AI Removed rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Removed rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Removed rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Removed rules for AI crawler Bytespider (ByteDance)
  • 🤖 AI Removed rules for AI crawler CCBot (Common Crawl)
  • … 13 more change(s)
view diff → snapshot
The Telegraph telegraph.co.uk modified +68 −1
  • 🤖 AI Added Allow: /bah-avis/usa-sports-road-trip-map for AI2Bot
  • 🤖 AI Added Allow: /bespoke-content/ for AI2Bot
  • 🤖 AI Added Allow: /betting/ for AI2Bot
  • 🤖 AI Added Allow: /brand-usa for AI2Bot
  • 🤖 AI Added Allow: /branded-content/ for AI2Bot
  • 🤖 AI Added Allow: /british-airways-holidays/cyprus-hotel-map for AI2Bot
  • … 45 more change(s)
view diff → snapshot
The Atlantic theatlantic.com modified +0 −14
  • Removed Disallow: /4624/TheAtlanticOnline/* for *
  • Removed Disallow: /4624/TheAtlanticOnline/* for AmazonAdBot
  • Removed Disallow: /4624/TheAtlanticOnline/* for Applebot
  • Removed Disallow: /4624/TheAtlanticOnline/* for ArcSpanScraper
  • Removed Disallow: /4624/TheAtlanticOnline/* for BingBot
  • 🤖 AI Removed Disallow: /4624/TheAtlanticOnline/* for ChatGPT-User
  • … 8 more change(s)
view diff → snapshot