NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
227this week
All changes 🤖 AI-crawler changes Hiding initial archives
Bar-Ilan University biu.ac.il removed +0 −207
  • robots.txt is no longer served (HTTP 404)
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/how-coinbase-used-code-connect-to-shrink-token-costs/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/how-we-built-generative-plugins-and-shaders/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-coinbase-used-code-connect-to-shrink-token-costs/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-we-built-generative-plugins-and-shaders/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/how-coinbase-used-code-connect-to-shrink-token-costs/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/how-we-built-generative-plugins-and-shaders/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Yandex yandex.com modified +2 −0
  • Added Disallow: /alice/chat/ for *
  • Added Disallow: /alice/chat/ for yandex
view diff → snapshot
DBLP dblp.org modified +71 −1
  • Blocked user-agent ‘AgentTimes’ entirely
  • Blocked user-agent ‘AI2Bot-DeepResearchEval’ entirely
  • Blocked user-agent ‘AIWebIndex’ entirely
  • Blocked user-agent ‘amazon-kendra’ entirely
  • Blocked user-agent ‘amazon-QBusiness’ entirely
  • Blocked user-agent ‘AmazonBuyForMe’ entirely
  • … 45 more change(s)
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Bar-Ilan University biu.ac.il removed +0 −207
  • robots.txt is no longer served (HTTP 404)
SAP sap.com modified +45 −8
  • 🤖 AI Added Disallow: /fetch for Amazonbot
  • 🤖 AI Added Disallow: /proxy for Amazonbot
  • 🤖 AI Added Disallow: /sharedlibs/ for Amazonbot
  • 🤖 AI Added Disallow: /fetch for ChatGPT-User
  • 🤖 AI Added Disallow: /proxy for ChatGPT-User
  • 🤖 AI Added Disallow: /sharedlibs/ for ChatGPT-User
  • … 12 more change(s)
view diff → snapshot
BP bp.com modified +0 −2
  • Removed Disallow: /en_us/united-states/home/referencing.html for *
  • Removed Disallow: /en_us/united-states/home/referencing.html for Algolia Crawler
view diff → snapshot