NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
Pinterest pinterest.com modified +81 −73
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261206_cayman_thanksgiving_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261208_hanukkah_day_4_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261209_hanukkah_day_5_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261212_hanukkah_day_8_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261212_last_day_of_chanukah_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261212_last_day_of_hanukkah_www.pinterest.com.xml
  • … 2 more change(s)
view diff → snapshot
Axel Springer axelspringer.com modified +6 −3
  • Added Disallow: (empty) for *
  • Removed Allow: /wp-admin/admin-ajax.php for *
  • Removed Disallow: /wp-admin/ for *
  • Added sitemap https://www.axelspringer.com/sitemap_index.xml
  • Removed sitemap https://www.axelspringer.com/wp-sitemap.xml
view diff → snapshot
Sysco sysco.com modified +0 −2
  • Removed Disallow: /sustainability/human-rights for *
  • Removed Disallow: /sysco-and-restaurant-depot-better-together for *
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Zebra Technologies zebra.com modified +215 −215
  • Added Disallow: */0003/zebra-it-security-video-portfolio-compliance-en-us.mp4 for *
  • Added Disallow: */3600-series-reference-components.html for *
  • Added Disallow: */5x5-end-user/* for *
  • Added Disallow: */5x5-partner/* for *
  • Added Disallow: */LabelsHero.html for *
  • Added Disallow: */ad-reference.html for *
  • … 45 more change(s)
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Amazon amazon.com modified +2 −0
  • Added Disallow: /shoptheshow/*&cat= for *
  • Added Disallow: /shoptheshow/*?cat= for *
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +7 −2
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /members/*/mem for *
  • Added Disallow: /providers/*/duals for *
  • Added Disallow: /providers/*/medicaid for *
  • Added Disallow: /providers/*/medicare for *
  • Removed Disallow: (empty) for *
  • … 1 more change(s)
view diff → snapshot
Arrow Electronics arrow.com removed +0 −116
  • robots.txt is no longer served (HTTP 404)
Humana humana.com modified +21 −0
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • Added rules for user-agent ‘Googlebot’
  • Added rules for user-agent ‘GoogleOther’
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Fresenius fresenius.com modified +6 −1
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
  • … 1 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +2 −7
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /members/*/mem for *
  • Removed Disallow: /providers/*/duals for *
  • Removed Disallow: /providers/*/medicaid for *
  • Removed Disallow: /providers/*/medicare for *
  • … 1 more change(s)
view diff → snapshot