NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
All changes 🤖 AI-crawler changes Hiding initial archives
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Snapchat snapchat.com modified +1 −0
  • Added sitemap https://www.snapchat.com/sitemaps/entertainment_funny/sitemap-index.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Barracuda barracuda.com created +28 −28
  • Added Disallow: /2QGvuWDJzNTl_zQtUa-kCYxc421SSSioP4rLS4V3Tic=.html for *
  • Removed Disallow: /ao4oVNcRKejUUUcGuNvfODmx1iIeG0WoFyjrrlAaARk=.html for *
  • Added Disallow: /2QGvuWDJzNTl_zQtUa-kCYxc421SSSioP4rLS4V3Tic=.html for DotBot
  • Removed Disallow: /ao4oVNcRKejUUUcGuNvfODmx1iIeG0WoFyjrrlAaARk=.html for DotBot
  • Added Disallow: /2QGvuWDJzNTl_zQtUa-kCYxc421SSSioP4rLS4V3Tic=.html for sistrix
  • Removed Disallow: /ao4oVNcRKejUUUcGuNvfODmx1iIeG0WoFyjrrlAaARk=.html for sistrix
  • … 2 more change(s)
view diff → snapshot
Ford Motor ford.com modified +2 −46
  • Removed all rules for user-agent ‘amazon-kendra’
  • Removed all rules for user-agent ‘amazon-QBusiness’
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Removed rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Removed rules for AI crawler Applebot-Extended (Apple)
  • Removed all rules for user-agent ‘bedrockbot’
  • … 26 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
DHL Group dhl.com modified +1 −5
  • Removed Disallow: /lu-de/home/globale-weiterleitung/hilfecenter/claims-complaints-compliments-form.html for *
  • Removed Disallow: /lu-en/home/global-forwarding/help-center/claims-complaints-compliments-form.html for *
  • Removed Disallow: /lu-fr/home/expediteur-mondial/centre-d-assistance/claims-complaints-compliments-form.html for *
  • Removed Disallow: /lu-fr/home/global-forwarding/help-center/claims-complaints-compliments-form.html for *
view diff → snapshot
Realtor realtor.com modified +3 −0
  • Added Disallow: /advice/draft/ for *
  • Added Disallow: /guides/draft/ for *
  • Added Disallow: /news/draft/ for *
view diff → snapshot