NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
All changes 🤖 AI-crawler changes Showing initial archives
ElevenLabs elevenlabs.io modified +93 −0
  • Added rules for user-agent ‘facebookexternalhit’
  • Added rules for user-agent ‘LinkedInBot’
  • Added rules for user-agent ‘WhatsApp’
view diff → snapshot
Figma Community figma.com modified +5 −0
  • 🤖 AI Added Allow: /agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/3-ways-product-designers-use-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/figma-agent-and-design-systems/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/workflow-lab-staying-in-the-flow-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/agent-credit-promo-project/$ for CHATGPT-User
  • 🤖 AI Added Allow: /agent/$ for Claude-SearchBot
  • … 19 more change(s)
view diff → snapshot
Iceland (Government.is) government.is modified +5 −0
  • Added Disallow: /page/e0a344e1-b8e8-4f99-873d-9f8cc3d3cb0c for *
  • Added Disallow: /topics/operations-and-state-assets/policy-on-public-services$ for *
  • Added Disallow: /topics/operations-and-state-assets/policy-on-public-services/$ for *
  • Added Disallow: /topics/operations-and-state-assets/policy-on-public-services/? for *
  • Added Disallow: /topics/operations-and-state-assets/policy-on-public-services? for *
view diff → snapshot
SoundCloud soundcloud.com modified +29 −60
  • Blocked user-agent ‘Brightbot’ entirely
  • 🤖 AI Blocked omgili (Webz.io) entirely
  • Blocked user-agent ‘omgilibot’ entirely
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Time time.com modified +0 −4
  • Removed sitemap https://time.com/archive-sitemaps.xml
  • Removed sitemap https://time.com/coupons/sitemap.xml
  • Removed sitemap https://time.com/sitemap-sections.xml
  • Removed sitemap https://time.com/sitemap-tags.xml
view diff → snapshot
SoundCloud soundcloud.com modified +60 −29
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Blocked cohere-ai (Cohere) entirely
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /legal/ai-add-on-promotional-offer/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/ai-add-on-promotional-offer/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /legal/ai-add-on-promotional-offer/$ for Claude-User
  • 🤖 AI Added Allow: /legal/ai-add-on-promotional-offer/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /legal/ai-add-on-promotional-offer/$ for PerplexityBot
view diff → snapshot
Bloomberg bloomberg.com modified +80 −0
  • 🤖 AI Added Allow: /africa/llms.txt$ for ChatGPT-User
  • 🤖 AI Added Allow: /africa/llms.txt/$ for ChatGPT-User
  • 🤖 AI Added Allow: /asia/llms.txt$ for ChatGPT-User
  • 🤖 AI Added Allow: /asia/llms.txt/$ for ChatGPT-User
  • 🤖 AI Added Allow: /canada/llms.txt$ for ChatGPT-User
  • 🤖 AI Added Allow: /canada/llms.txt/$ for ChatGPT-User
  • … 45 more change(s)
view diff → snapshot
Arrow Electronics arrow.com removed +0 −103
  • robots.txt is no longer served (HTTP 404)
Figma Community figma.com modified +3 −0
  • 🤖 AI Added Allow: /de-de/blog/what-it-takes-to-build-great-products-now/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/what-it-takes-to-build-great-products-now/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/what-it-takes-to-build-great-products-now/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/what-it-takes-to-build-great-products-now/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /es-es/blog/what-it-takes-to-build-great-products-now/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /fr-fr/blog/what-it-takes-to-build-great-products-now/$ for Claude-SearchBot
  • … 9 more change(s)
view diff → snapshot