The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1198files archived

2712snapshots

1543changes detected

232this week

All changes 🤖 AI-crawler changes Showing initial archives

Conagra Brands conagrabrands.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

General Mills generalmills.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Constellation Energy constellationenergy.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

NRG Energy nrg.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Vistra vistra.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

CenterPoint Energy centerpointenergy.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Ameren ameren.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

DTE Energy dteenergy.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Entergy entergy.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

FirstEnergy firstenergycorp.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Public Service Enterprise Group pseg.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Consolidated Edison conedison.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Edison International edison.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

PG&E pge.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Exelon exeloncorp.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Dominion Energy dominionenergy.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

NextEra Energy nexteraenergy.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Southern Company southerncompany.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Duke Energy duke-energy.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Sempra sempra.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Vattenfall vattenfall.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Oneok oneok.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Naturgy naturgy.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Williams williams.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

El Corte Ingles elcorteingles.es initial 1mo ago

First snapshot of robots.txt archived

snapshot

Kinder Morgan kindermorgan.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Leroy Merlin (Adeo) adeo.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

SLB slb.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Decathlon decathlon.com initial 1mo ago

First snapshot of robots.txt archived

snapshot

Halliburton halliburton.com initial 1mo ago

First snapshot of robots.txt archived

snapshot