NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2684snapshots
1514changes detected
226this week
All changes 🤖 AI-crawler changes Showing initial archives
TSMC tsmc.com initial
  • First snapshot of robots.txt archived
snapshot
SlideShare slideshare.net initial
  • First snapshot of robots.txt archived
snapshot
Scribd scribd.com initial
  • First snapshot of robots.txt archived
snapshot
Politico politico.com initial
  • First snapshot of robots.txt archived
snapshot
Pfizer pfizer.com initial
  • First snapshot of robots.txt archived
snapshot
Midjourney midjourney.com initial
  • First snapshot of robots.txt archived
snapshot
Michelin michelin.com initial
  • First snapshot of robots.txt archived
snapshot
Klarna klarna.com initial
  • First snapshot of robots.txt archived
snapshot
Imgur imgur.com initial
  • First snapshot of robots.txt archived
snapshot
Ferrari ferrari.com initial
  • First snapshot of robots.txt archived
snapshot
Ericsson ericsson.com initial
  • First snapshot of robots.txt archived
snapshot
E.ON eon.com initial
  • First snapshot of robots.txt archived
snapshot
DoorDash doordash.com initial
  • First snapshot of robots.txt archived
snapshot
CBS News cbsnews.com initial
  • First snapshot of robots.txt archived
snapshot
BlackRock blackrock.com initial
  • First snapshot of robots.txt archived
snapshot
Allianz allianz.com initial
  • First snapshot of robots.txt archived
snapshot
Goodyear Tire & Rubber goodyear.com modified +0 −40
  • Removed all rules for user-agent ‘ahrefsbot’
  • Removed all rules for user-agent ‘cazoodlebot’
  • Removed all rules for user-agent ‘dotbot/1.0’
  • Removed all rules for user-agent ‘gigabot’
  • Removed all rules for user-agent ‘mj12bot’
  • Removed all rules for user-agent ‘sogou inst spider’
  • … 20 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot