NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Hiding initial archives
Progressive progressive.com modified +3 −0
  • Added Disallow: /lifelanes/wp-content/ for *
  • Added Disallow: /lifelanes/wp-content/ for AdsBot-Google
  • Added Disallow: /lifelanes/wp-content/ for AdsBot-Google-Mobile
view diff → snapshot
Oncrawl oncrawl.com modified +2 −29
  • Removed all rules for user-agent ‘Googlebot’
  • Added Allow: /wp/wp-admin/admin-ajax.php for *
  • Added Disallow: /wp/wp-admin/ for *
  • Removed Disallow: /&sa=* for *
  • Removed Disallow: /*&sa=* for *
  • Removed Disallow: /.well-known* for *
  • … 9 more change(s)
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Nationwide nationwide.com modified +99 −1
  • Unblocked user-agent ‘*’
  • Added Disallow: /*.pdf$ for *
  • Added Disallow: /*ATCLID=* for *
  • Added Disallow: /*NWOSSi=* for *
  • Added Disallow: /*NWOSSiPromo=* for *
  • Added Disallow: /*NWOSSiPromoPos=* for *
  • … 45 more change(s)
view diff → snapshot
Nubank nubank.com.br modified +78 −84
  • Removed Disallow: /contratos/consignado-do-trabalhador for *
  • Removed Disallow: /contratos/consignado-do-trabalhador for DatoCmsSearchBot
view diff → snapshot
Nationwide nationwide.com modified +1 −99
  • Blocked user-agent ‘*’ entirely
  • Removed Disallow: /*.pdf$ for *
  • Removed Disallow: /*ATCLID=* for *
  • Removed Disallow: /*NWOSSi=* for *
  • Removed Disallow: /*NWOSSiPromo=* for *
  • Removed Disallow: /*NWOSSiPromoPos=* for *
  • … 45 more change(s)
view diff → snapshot
Arkose Labs arkoselabs.com modified +1 −2
  • Blocked user-agent ‘SiteAuditBot’ entirely
  • Removed Disallow: (empty) for SiteAuditBot
view diff → snapshot
Barracuda barracuda.com created +27 −27
  • Added Disallow: /GqS-MCUe_qfcMaNsUdAZhx3lvVU0SOEjrSLgx8DrQ9M=.html for *
  • Removed Disallow: /GivfqzeDvfdw9hMC07duOnQLNMB0MS0XyvQGw5MZSb8=.html for *
  • Added Disallow: /GqS-MCUe_qfcMaNsUdAZhx3lvVU0SOEjrSLgx8DrQ9M=.html for DotBot
  • Removed Disallow: /GivfqzeDvfdw9hMC07duOnQLNMB0MS0XyvQGw5MZSb8=.html for DotBot
  • Added Disallow: /GqS-MCUe_qfcMaNsUdAZhx3lvVU0SOEjrSLgx8DrQ9M=.html for sistrix
  • Removed Disallow: /GivfqzeDvfdw9hMC07duOnQLNMB0MS0XyvQGw5MZSb8=.html for sistrix
  • … 2 more change(s)
view diff → snapshot
Toyota toyota.com modified +1 −15
  • Removed Disallow: /espanol/vehicles/ for *
  • Removed Disallow: /vehicles/ for *
  • Removed Disallow: /espanol/vehicles/ for Applebot
  • Removed Disallow: /vehicles/ for Applebot
  • 🤖 AI Removed Disallow: /espanol/vehicles/ for ChatGPT-User
  • 🤖 AI Removed Disallow: /vehicles/ for ChatGPT-User
  • … 8 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap https://www.iqvia.com/image-sitemap.xml
  • Added sitemap https://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot