NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
Facebook facebook.com modified +3 −47
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • 🤖 AI Removed Allow: /ajax/bootloader-endpoint/ for Google-Extended
  • 🤖 AI Removed Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet for Google-Extended
  • 🤖 AI Removed Allow: /careers/ for Google-Extended
  • 🤖 AI Removed Allow: /safetycheck/ for Google-Extended
  • 🤖 AI Removed Disallow: /*/plugins/* for Google-Extended
  • … 40 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +29 −2
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /marketplace/az/* for *
  • Added Disallow: /marketplace/ca/* for *
  • Added Disallow: /marketplace/fl/* for *
  • Added Disallow: /marketplace/ga/* for *
  • Added Disallow: /marketplace/ia/* for *
  • … 23 more change(s)
view diff → snapshot
Sorbonne University sorbonne-universite.fr modified +6 −0
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
CNN cnn.com modified +2 −1
  • Blocked user-agent ‘ExaSearchBot’ entirely
  • Blocked user-agent ‘ExaUser’ entirely
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
view diff → snapshot
TotalEnergies totalenergies.com modified +1 −1
  • Added sitemap https://prd-backoffice.totalenergies.com/sitemap.xml
  • Removed sitemap https://totalenergies.com/sitemap.xml
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +2 −29
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /marketplace/az/* for *
  • Removed Disallow: /marketplace/ca/* for *
  • Removed Disallow: /marketplace/fl/* for *
  • Removed Disallow: /marketplace/ga/* for *
  • … 23 more change(s)
view diff → snapshot
Performance Food Group pfgc.com modified +7 −2
  • Added Allow: / for *
  • Removed Disallow: (empty) for *
  • Added sitemap https://pfgc.com/sitemap.xml
  • Removed sitemap https://www.pfgc.com/pfgc.xml
view diff → snapshot
Figma Community figma.com modified +3 −0
  • 🤖 AI Added Allow: /id-id/solusi/penghapus-latar-belakang-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /nl-nl/oplossingen/ai-achtergrond-verwijderaar/$ for CHATGPT-User
  • 🤖 AI Added Allow: /th-th/solutions/ai-background-remover/$ for CHATGPT-User
  • 🤖 AI Added Allow: /id-id/solusi/penghapus-latar-belakang-ai/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /nl-nl/oplossingen/ai-achtergrond-verwijderaar/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /th-th/solutions/ai-background-remover/$ for Claude-SearchBot
  • … 9 more change(s)
view diff → snapshot
Weather weather.com modified +10 −1
  • Added Disallow: /*&metric= for *
  • Added Disallow: /*&t= for *
  • Added Disallow: /*/hourbyhour/day- for *
  • Added Disallow: /*/localalerts?*&phenomena= for *
  • Added Disallow: /*/localalerts?phenomena= for *
  • Added Disallow: /*?metric= for *
  • … 1 more change(s)
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Pinterest pinterest.com modified +104 −84
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261127_presidents_day_observed_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261208_feast_of_the_immaculate_conception_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261224_washington_s_birthday_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261225_quaid_e_azam_day_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261226_christmas_second_day_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261226_day_after_christmas_christians_only_www.pinterest.com.xml
  • … 14 more change(s)
view diff → snapshot
Academia.edu academia.edu modified +55 −0
  • Added Disallow: /published_today_preview for APIs-Google
  • Added Disallow: /published_today_preview for Applebot
  • Added Disallow: /published_today_preview for ArchitextSpider
  • Added Disallow: /published_today_preview for baiduspider
  • Added Disallow: /published_today_preview for bingbot
  • Added Disallow: /published_today_preview for BingPreview
  • … 45 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +28 −3
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /marketplace/az/* for *
  • Added Disallow: /marketplace/ca/* for *
  • Added Disallow: /marketplace/fl/* for *
  • Added Disallow: /marketplace/ga/* for *
  • Added Disallow: /marketplace/ia/* for *
  • … 24 more change(s)
view diff → snapshot
ElevenLabs elevenlabs.io modified +0 −10
  • Removed Disallow: /nl$ for *
  • Removed Disallow: /nl/ for *
  • Removed Disallow: /nl$ for facebookexternalhit
  • Removed Disallow: /nl/ for facebookexternalhit
  • Removed Disallow: /nl$ for LinkedInBot
  • Removed Disallow: /nl/ for LinkedInBot
  • … 4 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +3 −28
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /marketplace/az/* for *
  • Removed Disallow: /marketplace/ca/* for *
  • Removed Disallow: /marketplace/fl/* for *
  • Removed Disallow: /marketplace/ga/* for *
  • … 24 more change(s)
view diff → snapshot