NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
227this week
All changes 🤖 AI-crawler changes Hiding initial archives
Figma Community figma.com modified +7 −6
  • 🤖 AI Added Allow: /de-de/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Guyana (Office of the President) op.gov.gy modified +0 −5
  • Removed Allow: /wp-admin/admin-ajax.php for *
  • Removed Disallow: /wp-admin/ for *
  • Removed sitemap https://op.gov.gy/news-sitemap.xml
  • Removed sitemap https://op.gov.gy/sitemap.xml
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /es-la/blog/source-material-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-thom-bettridge/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /es-la/blog/source-material-thom-bettridge/$ for Claude-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-thom-bettridge/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /es-la/blog/source-material-thom-bettridge/$ for PerplexityBot
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Reuters reuters.com modified +0 −1
  • Removed Disallow: /jp/ for AASA-Bot
  • Removed Disallow: /jp/ for ADmantX
  • Removed Disallow: /jp/ for AdmiralBot
  • Removed Disallow: /jp/ for AdsBot-Google
  • Removed Disallow: /jp/ for AdsBot-Google-Mobile
  • Removed Disallow: /jp/ for AmazonAdBot
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Lululemon Athletica lululemon.com modified +9 −6
  • 🤖 AI Blocked Meta-WebIndexer (Meta) entirely
  • Added Allow: /*/c/*/*?*icid=* for *
  • Added Allow: /c/*/*?*icid=* for *
  • Added Disallow: /*/c/*/*z*z*z* for *
  • Added Disallow: /*/search?Ntt= for *
  • Added Disallow: /c/*/*z*z*z* for *
  • … 4 more change(s)
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Cemex cemex.com modified +46 −2
  • Added Allow: /documents/ for *
  • Added Allow: /favicon.ico for *
  • Added Allow: /image/* for *
  • Added Allow: /o/web/image/* for *
  • Added Allow: /sitemap.xml for *
  • Added Disallow: */web for *
  • … 32 more change(s)
view diff → snapshot