NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
All changes 🤖 AI-crawler changes Showing initial archives
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
MetLife metlife.com modified +8 −2
  • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
  • Added Disallow: https://metlife-qa1-65.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
MetLife metlife.com modified +2 −8
  • Added Disallow: /investments$ for *
  • Added Disallow: /investments/ for *
  • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Nubank nubank.com.br modified +0 −6
  • Removed Disallow: /nu-design-day for *
  • Removed Disallow: /nu-design-day for DatoCmsSearchBot
view diff → snapshot
Barracuda barracuda.com created +28 −28
  • Added Disallow: /AhL8dV0Pb0uaG5WQP3DgESS1GUB2wnrlikHc2eAdbUE=.html for *
  • Removed Disallow: /70hkWk5EPQLRDbV9Q7189DJB_y14lTx9BS0qLGJVSZA=.html for *
  • Added Disallow: /AhL8dV0Pb0uaG5WQP3DgESS1GUB2wnrlikHc2eAdbUE=.html for DotBot
  • Removed Disallow: /70hkWk5EPQLRDbV9Q7189DJB_y14lTx9BS0qLGJVSZA=.html for DotBot
  • Added Disallow: /AhL8dV0Pb0uaG5WQP3DgESS1GUB2wnrlikHc2eAdbUE=.html for sistrix
  • Removed Disallow: /70hkWk5EPQLRDbV9Q7189DJB_y14lTx9BS0qLGJVSZA=.html for sistrix
  • … 2 more change(s)
view diff → snapshot
Nubank nubank.com.br modified +36 −30
  • Added Disallow: /nu-design-day for *
  • Added Disallow: /nu-design-day for DatoCmsSearchBot
view diff → snapshot