NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
All changes 🤖 AI-crawler changes Hiding initial archives
World Kinect world-kinect.com modified +4 −68
  • Added Allow: /wp-admin/admin-ajax.php for *
  • Added Disallow: /wp-admin/ for *
  • Removed Allow: /core/*.css$ for *
  • Removed Allow: /core/*.css? for *
  • Removed Allow: /core/*.gif for *
  • Removed Allow: /core/*.jpeg for *
  • … 44 more change(s)
view diff → snapshot
Adorama adorama.com modified +1 −3
  • Added Disallow: /upload/*.m3u8 for *
  • Removed Disallow: /alc/category/photography/page/ for *
  • Removed Disallow: /alc/wp-json/ for *
  • Removed Disallow: /nb/* for *
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Argentina (Argentina.gob.ar) argentina.gob.ar modified +6 −2
  • Added Disallow: /?q=servicios/ for *
  • Added Disallow: /?q=tramitar/ for *
  • Added Disallow: /buscar for *
  • Added Disallow: /search for *
  • Added Disallow: /servicios for *
  • Added Disallow: /tramitar for *
  • … 1 more change(s)
view diff → snapshot
Proofpoint proofpoint.com modified +3 −0
  • Added Disallow: /sites/default/files/white-papers/pfpt-au-wp-securing-and-governing-AI-in-the-modern-enterprise-guide.pdf for *
  • Added Disallow: /sites/default/files/white-papers/pfpt-kr-wp-securing-and-governing-AI-in-the-modern-enterprise-guide.pdf for *
  • Added Disallow: /sites/default/files/white-papers/pfpt-uk-wp-securing-and-governing-AI-in-the-modern-enterprise-guide.pdf for *
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Arrow Electronics arrow.com created +6 −5
  • Removed Disallow: *%3Fregion* for *
  • Removed Disallow: */arrow-dxp/language-masters/ for *
  • Removed Disallow: *?region* for *
  • Removed Disallow: *utm_source=* for *
  • Removed Disallow: /content/arrow-dxp/ for *
view diff → snapshot
Figma Community figma.com modified +21 −1
  • 🤖 AI Added Allow: /resource-library/agentic-ai-for-content-creation/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/ai-cinematography/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/ai-short-film-prompts/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/color-grading/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/compositing/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/lip-sync-animation/$ for CHATGPT-User
  • … 34 more change(s)
view diff → snapshot
Slate slate.com modified +70 −1
  • Blocked user-agent ‘AgentDataBot’ entirely
  • Added rules for user-agent ‘AutotouchCompanyCrawler’
  • Blocked user-agent ‘BeansLLM-CorpusBot’ entirely
  • Blocked user-agent ‘BixelBot’ entirely
  • Added rules for user-agent ‘CandidOrgWebCheck’
  • Added rules for user-agent ‘centryx-domain-mill’
  • … 45 more change(s)
view diff → snapshot
Kimberly-Clark kimberly-clark.com modified +109 −2
  • Added rules for user-agent ‘AndiBot’
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • Added rules for user-agent ‘Applebot’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Added rules for user-agent ‘Bingbot’
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • … 45 more change(s)
view diff → snapshot