NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Hiding initial archives
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Oncrawl oncrawl.com modified +29 −2
  • Added rules for user-agent ‘Googlebot’
  • Added Disallow: /&sa=* for *
  • Added Disallow: /*&sa=* for *
  • Added Disallow: /.well-known* for *
  • Added Disallow: /?author=* for *
  • Added Disallow: /?s=* for *
  • … 9 more change(s)
view diff → snapshot