NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Showing initial archives
Humana humana.com modified +2 −0
  • Added sitemap https://www.humana.com/sitemap-ai-index.xml
  • Added sitemap https://www.humana.com/sitemap-sb-pdfs.xml
view diff → snapshot
Nubank nubank.com.br modified +6 −0
  • Added Disallow: /shopee for *
  • Added Disallow: /shopee for DatoCmsSearchBot
view diff → snapshot
Al Jazeera aljazeera.com modified +2 −1
  • Added Disallow: /*&update= for *
  • Added Disallow: /*?update= for *
  • Removed Disallow: /*/liveblog/*?*update=* for *
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/training-a-figma-fluent-editing-model/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/training-a-figma-fluent-editing-model/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/training-a-figma-fluent-editing-model/$ for Claude-User
  • 🤖 AI Added Allow: /blog/training-a-figma-fluent-editing-model/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/training-a-figma-fluent-editing-model/$ for PerplexityBot
view diff → snapshot
Newsweek newsweek.com modified +6 −2
  • 🤖 AI Blocked Perplexity-User (Perplexity) entirely
  • Blocked user-agent ‘Shap-User’ entirely
  • Blocked user-agent ‘ShapBot’ entirely
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
view diff → snapshot
Delta Air Lines delta.com modified +67 −12
  • Added rules for user-agent ‘ai2-crawler’
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • … 31 more change(s)
view diff → snapshot
Bild bild.de modified +16 −0
  • Added Disallow: /casinos/go/ for *
  • Added Disallow: /dating/go/ for *
  • Added Disallow: /energieloesungen/go/ for *
  • Added Disallow: /energieloesungen/social/go/ for *
  • Added Disallow: /energieloesungen/vergleich/go/ for *
  • Added Disallow: /games/go/ for *
  • … 10 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /customers/how-govtech-singapore-teams-use-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/how-govtech-singapore-teams-use-figma/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /customers/how-govtech-singapore-teams-use-figma/$ for Claude-User
  • 🤖 AI Added Allow: /customers/how-govtech-singapore-teams-use-figma/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /customers/how-govtech-singapore-teams-use-figma/$ for PerplexityBot
view diff → snapshot
Instagram instagram.com modified +1 −0
  • Added sitemap https://www.instagram.com/sitemap/ig_seo_historical_reels_mvp_video_rows.xml.gz
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +2 −29
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /marketplace/az/* for *
  • Removed Disallow: /marketplace/ca/* for *
  • Removed Disallow: /marketplace/fl/* for *
  • Removed Disallow: /marketplace/ga/* for *
  • … 23 more change(s)
view diff → snapshot