NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Hiding initial archives
Condé Nast condenast.com removed +0 −57
  • robots.txt is no longer served (HTTP 404)
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
ChatGPT chatgpt.com modified +3 −2
  • Added Allow: /m/ for *
  • Added Disallow: /s/ for AdsBot-Google
  • Removed Disallow: /s/cb_ for AdsBot-Google
  • Added Disallow: /s/ for AdsBot-Google-Mobile
  • Removed Disallow: /s/cb_ for AdsBot-Google-Mobile
view diff → snapshot
CME Group cmegroup.com modified +1 −2
  • Added sitemap https://www.cmegroup.com/sitemap_index.xml
  • Removed sitemap https://www.cmegroup.com/eds/sitemap.xml
  • Removed sitemap https://www.cmegroup.com/sitemap.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
MetLife metlife.com modified +0 −1
  • Removed sitemap https://www.metlife.com/investments/global/sitemap.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot