NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
All changes 🤖 AI-crawler changes Hiding initial archives
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Dell Technologies dell.com modified +3 −17
  • Removed Disallow: */gaming/alienware for *
  • Removed Disallow: */sac/*/*/*?appliedRefinements=* for *
  • Removed Disallow: */shop/*/sac/* for *
  • Removed Disallow: */shop/accessories$ for *
  • Removed Disallow: */shop/alienware/gaming$ for *
  • Removed Disallow: */shop/alienware/gaming/* for *
  • … 10 more change(s)
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Dell Technologies dell.com modified +36 −1
  • Added Disallow: */ar/*/*?appliedRefinements=* for *
  • Added Disallow: */campaign/lp/ for *
  • Added Disallow: */gaming/alienware for *
  • Added Disallow: */learn/lp/ for *
  • Added Disallow: */legal/lp/ for *
  • Added Disallow: */lp/ for *
  • … 27 more change(s)
view diff → snapshot
Check Point checkpoint.com modified +1 −1
  • Added sitemap https://www.checkpoint.com/sitemap_index.xml
  • Removed sitemap https://www.checkpoint.com/sitemap.xml
view diff → snapshot
Dell Technologies dell.com modified +5 −178
  • Removed Allow: /da-dk/perspectives/ for *
  • Removed Allow: /de-at/perspectives/ for *
  • Removed Allow: /de-ch/perspectives/ for *
  • Removed Allow: /de-de/perspectives/ for *
  • Removed Allow: /en-au/perspectives/ for *
  • Removed Allow: /en-ca/perspectives/ for *
  • … 45 more change(s)
view diff → snapshot
Academia.edu academia.edu modified +0 −2
  • 🤖 AI Removed Allow: /3 for Google-Extended
  • 🤖 AI Removed Allow: /9 for Google-Extended
view diff → snapshot
GitHub github.com modified +90 −45
  • Added Disallow: /*&author=* for *
  • Added Disallow: /*&plan=* for *
  • Added Disallow: /*&q=* for *
  • Added Disallow: /*&ref_cta=* for *
  • Added Disallow: /*&ref_loc=* for *
  • Added Disallow: /*&ref_page=* for *
  • … 45 more change(s)
view diff → snapshot