NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1204files archived
3937snapshots
2782changes detected
216this week
All changes 🤖 AI-crawler changes Showing initial archives
Dell Technologies dell.com modified +36 −1
  • Added Disallow: */ar/*/*?appliedRefinements=* for *
  • Added Disallow: */campaign/lp/ for *
  • Added Disallow: */gaming/alienware for *
  • Added Disallow: */learn/lp/ for *
  • Added Disallow: */legal/lp/ for *
  • Added Disallow: */lp/ for *
  • … 27 more change(s)
view diff → snapshot
Check Point checkpoint.com modified +1 −1
  • Added sitemap https://www.checkpoint.com/sitemap_index.xml
  • Removed sitemap https://www.checkpoint.com/sitemap.xml
view diff → snapshot
Dell Technologies dell.com modified +5 −178
  • Removed Allow: /da-dk/perspectives/ for *
  • Removed Allow: /de-at/perspectives/ for *
  • Removed Allow: /de-ch/perspectives/ for *
  • Removed Allow: /de-de/perspectives/ for *
  • Removed Allow: /en-au/perspectives/ for *
  • Removed Allow: /en-ca/perspectives/ for *
  • … 45 more change(s)
view diff → snapshot
Academia.edu academia.edu modified +0 −2
  • 🤖 AI Removed Allow: /3 for Google-Extended
  • 🤖 AI Removed Allow: /9 for Google-Extended
view diff → snapshot
GitHub github.com modified +90 −45
  • Added Disallow: /*&author=* for *
  • Added Disallow: /*&plan=* for *
  • Added Disallow: /*&q=* for *
  • Added Disallow: /*&ref_cta=* for *
  • Added Disallow: /*&ref_loc=* for *
  • Added Disallow: /*&ref_page=* for *
  • … 45 more change(s)
view diff → snapshot
Indeed indeed.com modified +33 −17
  • Removed all rules for user-agent ‘Bingbot’
  • Added Disallow: /Zeige-Job for *
  • Added Disallow: /generate-question for *
  • Added Disallow: /hire/ats-partners for *
  • Added Disallow: /hire/o/ats-partners for *
  • Added Disallow: /m/viewjob for *
  • … 45 more change(s)
view diff → snapshot
Fortra fortra.com modified +7 −0
  • Added rules for user-agent ‘SemrushBot’
  • Added rules for user-agent ‘SiteAuditBot’
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Commonwealth Bank commbank.com.au modified +4 −1
  • Added Allow: /etc/designs/commbank/favicon.ico for *
  • 🤖 AI Added Allow: /digital/home-loans/borrowing-power-calculator for Amazonbot
  • 🤖 AI Added Allow: /etc/designs/commbank/favicon.ico for Amazonbot
  • 🤖 AI Added Allow: /digital/home-loans/borrowing-power-calculator for Applebot-Extended
  • 🤖 AI Added Allow: /etc/designs/commbank/favicon.ico for Applebot-Extended
  • 🤖 AI Added Allow: /digital/home-loans/borrowing-power-calculator for Bytespider
  • … 29 more change(s)
view diff → snapshot