NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
227this week
All changes 🤖 AI-crawler changes Hiding initial archives
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Hacker News news.ycombinator.com modified +0 −4
  • Removed Disallow: /favex? for *
  • Removed Disallow: /loginx for *
  • Removed Disallow: /replyx? for *
  • Removed Disallow: /votex? for *
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Bar-Ilan University biu.ac.il removed +0 −207
  • robots.txt is no longer served (HTTP 404)
Proofpoint proofpoint.com modified +8 −0
  • Added Disallow: /sites/default/files/white-papers/pfpt-au-wp-securing-ai-agents-guide.pdf for *
  • Added Disallow: /sites/default/files/white-papers/pfpt-br-wp-securing-ai-agents-guide.pdf for *
  • Added Disallow: /sites/default/files/white-papers/pfpt-de-wp-securing-ai-agents-guide.pdf for *
  • Added Disallow: /sites/default/files/white-papers/pfpt-es-wp-securing-ai-agents-guide.pdf for *
  • Added Disallow: /sites/default/files/white-papers/pfpt-fr-wp-securing-ai-agents-guide.pdf for *
  • Added Disallow: /sites/default/files/white-papers/pfpt-it-wp-securing-ai-agents-guide.pdf for *
  • … 2 more change(s)
view diff → snapshot
Siemens siemens.com removed +0 −6
  • robots.txt is no longer served (HTTP 404)
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Canva canva.com modified +3 −0
  • 🤖 AI Added Allow: /*/ai-connector/ for anthropic-ai
  • 🤖 AI Added Allow: /ai-connector/ for anthropic-ai
  • 🤖 AI Added Allow: /developers/ for anthropic-ai
  • 🤖 AI Added Allow: /*/ai-connector/ for Applebot-Extended
  • 🤖 AI Added Allow: /ai-connector/ for Applebot-Extended
  • 🤖 AI Added Allow: /developers/ for Applebot-Extended
  • … 12 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /webinars/code-connect-to-the-test-coinbase/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/code-connect-to-the-test-coinbase/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /webinars/code-connect-to-the-test-coinbase/$ for Claude-User
  • 🤖 AI Added Allow: /webinars/code-connect-to-the-test-coinbase/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /webinars/code-connect-to-the-test-coinbase/$ for PerplexityBot
view diff → snapshot
Coursera coursera.org modified +260 −2
  • Added Allow: /courses?*&page=1$ for *
  • Added Allow: /courses?*&page=1& for *
  • Added Allow: /courses?*&page=2$ for *
  • Added Allow: /courses?*&page=2& for *
  • Added Allow: /courses?*&page=3$ for *
  • Added Allow: /courses?*&page=3& for *
  • … 45 more change(s)
view diff → snapshot