NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2625snapshots
1453changes detected
242this week
All changes 🤖 AI-crawler changes Hiding initial archives
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*
  • Added rules for user-agent ‘Mediapartners-Google
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*
  • Removed all rules for user-agent ‘Mediapartners-Google
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
Estee Lauder elcompanies.com modified +12 −0
  • Added Disallow: /country-by-country-elco for *
  • Added Disallow: /country-by-country-romania for *
  • Added Disallow: /en-gb/country-by-country-elco for *
  • Added Disallow: /en-gb/country-by-country-romania for *
  • Added Disallow: /en/country-by-country-elco for *
  • Added Disallow: /en/country-by-country-romania for *
  • … 6 more change(s)
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml