NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1160sites tracked
1066files archived
1106snapshots
46changes detected
46this week
All changes 🤖 AI-crawler changes Showing initial archives
Visa visa.com initial
  • First snapshot of robots.txt archived
snapshot
Infineon infineon.com initial
  • First snapshot of robots.txt archived
snapshot
Prosus prosus.com initial
  • First snapshot of robots.txt archived
snapshot
USAA usaa.com initial
  • First snapshot of robots.txt archived
snapshot
Adyen adyen.com initial
  • First snapshot of robots.txt archived
snapshot
Spotify spotify.com initial
  • First snapshot of robots.txt archived
snapshot
Cummins cummins.com initial
  • First snapshot of robots.txt archived
snapshot
Mercadona mercadona.es initial
  • First snapshot of robots.txt archived
snapshot
Starbucks starbucks.com initial
  • First snapshot of robots.txt archived
snapshot
DNB dnb.no initial
  • First snapshot of robots.txt archived
snapshot
Metro AG metroag.de initial
  • First snapshot of robots.txt archived
snapshot
3M 3m.com initial
  • First snapshot of robots.txt archived
snapshot
Heineken heineken.com initial
  • First snapshot of robots.txt archived
snapshot
Tesco tescoplc.com initial
  • First snapshot of robots.txt archived
snapshot