NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2723snapshots
1554changes detected
243this week
All changes 🤖 AI-crawler changes Showing initial archives
Adyen adyen.com initial
  • First snapshot of robots.txt archived
snapshot
Spotify spotify.com initial
  • First snapshot of robots.txt archived
snapshot
Cummins cummins.com initial
  • First snapshot of robots.txt archived
snapshot
Mercadona mercadona.es initial
  • First snapshot of robots.txt archived
snapshot
Starbucks starbucks.com initial
  • First snapshot of robots.txt archived
snapshot
DNB dnb.no initial
  • First snapshot of robots.txt archived
snapshot
REWE Group rewe-group.com initial
  • First snapshot of robots.txt archived
snapshot
Metro AG metroag.de initial
  • First snapshot of robots.txt archived
snapshot
3M 3m.com initial
  • First snapshot of robots.txt archived
snapshot
Heineken heineken.com initial
  • First snapshot of robots.txt archived
snapshot
Salesforce salesforce.com initial
  • First snapshot of robots.txt archived
snapshot
Tesco tescoplc.com initial
  • First snapshot of robots.txt archived
snapshot
Sodexo sodexo.com initial
  • First snapshot of robots.txt archived
snapshot
Accor group.accor.com initial
  • First snapshot of robots.txt archived
snapshot
Qualcomm qualcomm.com initial
  • First snapshot of robots.txt archived
snapshot
Ryanair ryanair.com initial
  • First snapshot of robots.txt archived
snapshot
IAG iairgroup.com initial
  • First snapshot of robots.txt archived
snapshot