NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
All changes 🤖 AI-crawler changes Hiding initial archives
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
Ahrefs ahrefs.com modified +1 −2
  • Added Allow: /pt/blog for *
  • Removed Disallow: /*/new-blog for *
  • Removed Disallow: /new-blog for *
Yandex yandex.ru modified +2 −0
  • Added Disallow: /archive/search for Googlebot
  • Added sitemap https://yandex.ru/archive/sitemap.xml