NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Showing initial archives
Nextdoor nextdoor.com modified +24 −0
  • Added rules for user-agent ‘bingbot’
  • Added rules for user-agent ‘DuckDuckBot’
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
USA Today usatoday.com modified +12 −0
  • Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT Operator
  • Added Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT Operator
  • 🤖 AI Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT-User
  • 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT-User
  • 🤖 AI Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for Google-Extended
  • 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for Google-Extended
  • … 6 more change(s)
view diff → snapshot
Progressive progressive.com modified +3 −0
  • Added Disallow: /lifelanes/wp-content/ for *
  • Added Disallow: /lifelanes/wp-content/ for AdsBot-Google
  • Added Disallow: /lifelanes/wp-content/ for AdsBot-Google-Mobile
view diff → snapshot
Oncrawl oncrawl.com modified +2 −29
  • Removed all rules for user-agent ‘Googlebot’
  • Added Allow: /wp/wp-admin/admin-ajax.php for *
  • Added Disallow: /wp/wp-admin/ for *
  • Removed Disallow: /&sa=* for *
  • Removed Disallow: /*&sa=* for *
  • Removed Disallow: /.well-known* for *
  • … 9 more change(s)
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Nationwide nationwide.com modified +99 −1
  • Unblocked user-agent ‘*’
  • Added Disallow: /*.pdf$ for *
  • Added Disallow: /*ATCLID=* for *
  • Added Disallow: /*NWOSSi=* for *
  • Added Disallow: /*NWOSSiPromo=* for *
  • Added Disallow: /*NWOSSiPromoPos=* for *
  • … 45 more change(s)
view diff → snapshot
Nubank nubank.com.br modified +78 −84
  • Removed Disallow: /contratos/consignado-do-trabalhador for *
  • Removed Disallow: /contratos/consignado-do-trabalhador for DatoCmsSearchBot
view diff → snapshot
Nationwide nationwide.com modified +1 −99
  • Blocked user-agent ‘*’ entirely
  • Removed Disallow: /*.pdf$ for *
  • Removed Disallow: /*ATCLID=* for *
  • Removed Disallow: /*NWOSSi=* for *
  • Removed Disallow: /*NWOSSiPromo=* for *
  • Removed Disallow: /*NWOSSiPromoPos=* for *
  • … 45 more change(s)
view diff → snapshot
Arkose Labs arkoselabs.com modified +1 −2
  • Blocked user-agent ‘SiteAuditBot’ entirely
  • Removed Disallow: (empty) for SiteAuditBot
view diff → snapshot