NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5446snapshots
4311changes detected
246this week
All changes 🤖 AI-crawler changes Showing initial archives
Infineon infineon.com initial
  • First snapshot of robots.txt archived
snapshot
Prosus prosus.com initial
  • First snapshot of robots.txt archived
snapshot
USAA usaa.com initial
  • First snapshot of robots.txt archived
snapshot
Adyen adyen.com initial
  • First snapshot of robots.txt archived
snapshot
Spotify spotify.com initial
  • First snapshot of robots.txt archived
snapshot
Cummins cummins.com initial
  • First snapshot of robots.txt archived
snapshot
Mercadona mercadona.es initial
  • First snapshot of robots.txt archived
snapshot
Starbucks starbucks.com initial
  • First snapshot of robots.txt archived
snapshot
DNB dnb.no initial
  • First snapshot of robots.txt archived
snapshot
REWE Group rewe-group.com initial
  • First snapshot of robots.txt archived
snapshot
Metro AG metroag.de initial
  • First snapshot of robots.txt archived
snapshot
3M 3m.com initial
  • First snapshot of robots.txt archived
snapshot
Heineken heineken.com initial
  • First snapshot of robots.txt archived
snapshot
Salesforce salesforce.com initial
  • First snapshot of robots.txt archived
snapshot
Tesco tescoplc.com initial
  • First snapshot of robots.txt archived
snapshot
Sodexo sodexo.com initial
  • First snapshot of robots.txt archived
snapshot
Accor group.accor.com initial
  • First snapshot of robots.txt archived
snapshot