NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Hiding initial archives
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Eli Lilly lilly.com modified +3 −0
  • Added sitemap https://www.lilly.com/jp/henzutsu-karuta/sitemap.xml
  • Added sitemap https://www.lilly.com/jp/locj/sitemap.xml
  • Added sitemap https://www.lilly.com/jp/sitemap.xml
view diff → snapshot
ExtraHop extrahop.com modified +1 −1
  • Added sitemap https://www.extrahop.com/sitemap.xml
  • Removed sitemap https://www.extrahop.com/api/sitemap.xml
view diff → snapshot
The Economist economist.com modified +16 −0
  • 🤖 AI Added Allow: /syndication$ for ChatGPT-User
  • 🤖 AI Added Allow: /syndication/ for ChatGPT-User
  • 🤖 AI Added Allow: /syndication$ for Claude-SearchBot
  • 🤖 AI Added Allow: /syndication/ for Claude-SearchBot
  • 🤖 AI Added Allow: /syndication$ for Claude-User
  • 🤖 AI Added Allow: /syndication/ for Claude-User
  • … 10 more change(s)
view diff → snapshot
Valero Energy valero.com modified +6 −0
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
view diff → snapshot
Vox vox.com modified +100 −0
  • Added Allow: /strategist for A6-Indexer
  • Added Allow: /strategist for AhrefsBot
  • 🤖 AI Added Allow: /strategist for AI2Bot
  • Added Allow: /strategist for AI2Bot-DeepResearchEval
  • Added Allow: /strategist for Ai2Bot-Dolma
  • Added Allow: /strategist for aiHitBot
  • … 45 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Bild bild.de modified +0 −16
  • Removed Disallow: /casinos/go/ for *
  • Removed Disallow: /dating/go/ for *
  • Removed Disallow: /energieloesungen/go/ for *
  • Removed Disallow: /energieloesungen/social/go/ for *
  • Removed Disallow: /energieloesungen/vergleich/go/ for *
  • Removed Disallow: /games/go/ for *
  • … 10 more change(s)
view diff → snapshot
Westpac westpac.com.au modified +17 −27
  • Added Allow: / for *
  • Added Disallow: /test-aem-1817/ for *
  • Added Disallow: /test/ for *
  • Removed Disallow: /Forms/ for *
  • Removed Disallow: /business-banking/business-banking-online-sign-in-out/signin/ for *
  • Removed Disallow: /business-banking/merchant-services/mobile-payments/mobilepayway/ for *
  • … 18 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot