NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Showing initial archives
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +94 −0
  • 🤖 AI Added Allow: /blog/4-team-rituals-that-drive-ai-adoption/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/sightlines-issue-no3/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/legal/copyright-and-ip-policy/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/legal/core-features/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/legal/data-portability-and-deletion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/legal/impressum/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Humana humana.com modified +21 −0
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • Added rules for user-agent ‘Googlebot’
  • Added rules for user-agent ‘GoogleOther’
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /type-scale-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /type-scale-generator/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /type-scale-generator/$ for Claude-User
  • 🤖 AI Added Allow: /type-scale-generator/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /type-scale-generator/$ for PerplexityBot
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /ja-jp/reports/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/reports/2026-ai-report/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /ja-jp/reports/2026-ai-report/$ for Claude-User
  • 🤖 AI Added Allow: /ja-jp/reports/2026-ai-report/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /ja-jp/reports/2026-ai-report/$ for PerplexityBot
view diff → snapshot
BBC bbc.com modified +6 −4
  • 🤖 AI Added Allow: /storyworks for OAI-SearchBot
  • 🤖 AI Added Allow: /storyworks/ for OAI-SearchBot
view diff → snapshot
Arrow Electronics arrow.com created +13 −0
  • 🤖 AI Added Allow: /globalecs/* for anthropic-ai
  • 🤖 AI Added Allow: /globalecs/* for CCBot
  • 🤖 AI Added Allow: /globalecs/* for ChatGPT-User
  • 🤖 AI Added Allow: /globalecs/* for ClaudeBot
  • 🤖 AI Added Allow: /globalecs/* for Google-Extended
  • 🤖 AI Added Allow: /globalecs/* for GPTBot
  • … 1 more change(s)
view diff → snapshot
Mondelez International mondelezinternational.com modified +55 −0
  • Added rules for user-agent ‘bingbot’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Google-Extended (Google)
  • … 5 more change(s)
view diff → snapshot
Canva canva.com modified +4 −0
  • 🤖 AI Added Allow: /tr_tr/grafik/ for anthropic-ai
  • 🤖 AI Added Allow: /tr_tr/ozellikler/ for anthropic-ai
  • 🤖 AI Added Allow: /tr_tr/sema/ for anthropic-ai
  • 🤖 AI Added Allow: /tr_tr/yapma/ for anthropic-ai
  • 🤖 AI Added Allow: /tr_tr/grafik/ for Applebot-Extended
  • 🤖 AI Added Allow: /tr_tr/ozellikler/ for Applebot-Extended
  • … 18 more change(s)
view diff → snapshot
Indeed indeed.com modified +2 −0
  • Added Disallow: /promo/ for *
  • 🤖 AI Added Disallow: /promo/ for ChatGPT-User
  • 🤖 AI Added Disallow: /promo/ for Claude-SearchBot
  • 🤖 AI Added Disallow: /promo/ for Claude-User
  • 🤖 AI Added Disallow: /promo/ for DuckAssistBot
  • Added Disallow: /promo/ for DuckDuckBot
  • … 16 more change(s)
view diff → snapshot
Figma Community figma.com modified +0 −67
  • 🤖 AI Removed Allow: /de-de/legal/copyright-and-ip-policy/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/legal/core-features/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/legal/data-portability-and-deletion/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/legal/impressum/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/legal/shared-responsibility-security-model/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /fr-fr/legal/copyright-and-ip-policy/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Instagram instagram.com modified +77 −21
  • 🤖 AI Unblocked Amazonbot (Amazon)
  • 🤖 AI Added Allow: /places/c/ for Amazonbot
  • 🤖 AI Added Disallow: /accounts/login/*?next= for Amazonbot
  • 🤖 AI Added Disallow: /ajax/ for Amazonbot
  • 🤖 AI Added Disallow: /client_error/ for Amazonbot
  • 🤖 AI Added Disallow: /direct/ for Amazonbot
  • … 45 more change(s)
view diff → snapshot
NetApp netapp.com modified +82 −27
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • … 30 more change(s)
view diff → snapshot
Quora quora.com modified +5 −0
  • 🤖 AI Added Allow: /*Drinking-Water for OAI-SearchBot
  • 🤖 AI Added Allow: /*Drinking-water for OAI-SearchBot
  • 🤖 AI Added Allow: /*drinking-Water for OAI-SearchBot
  • 🤖 AI Added Allow: /*drinking-water for OAI-SearchBot
view diff → snapshot
Figma Community figma.com modified +1 −1
  • 🤖 AI Added Allow: /legal/figma-agent-credit-promo/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /legal/agent-credit-promo-project/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/figma-agent-credit-promo/$ for Claude-SearchBot
  • 🤖 AI Removed Allow: /legal/agent-credit-promo-project/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /legal/figma-agent-credit-promo/$ for Claude-User
  • 🤖 AI Removed Allow: /legal/agent-credit-promo-project/$ for Claude-User
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +5 −0
  • 🤖 AI Added Allow: /agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/3-ways-product-designers-use-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/figma-agent-and-design-systems/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/workflow-lab-staying-in-the-flow-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/agent-credit-promo-project/$ for CHATGPT-User
  • 🤖 AI Added Allow: /agent/$ for Claude-SearchBot
  • … 19 more change(s)
view diff → snapshot
SoundCloud soundcloud.com modified +29 −60
  • Blocked user-agent ‘Brightbot’ entirely
  • 🤖 AI Blocked omgili (Webz.io) entirely
  • Blocked user-agent ‘omgilibot’ entirely
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
SoundCloud soundcloud.com modified +60 −29
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Blocked cohere-ai (Cohere) entirely
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /legal/ai-add-on-promotional-offer/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/ai-add-on-promotional-offer/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /legal/ai-add-on-promotional-offer/$ for Claude-User
  • 🤖 AI Added Allow: /legal/ai-add-on-promotional-offer/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /legal/ai-add-on-promotional-offer/$ for PerplexityBot
view diff → snapshot
Bloomberg bloomberg.com modified +80 −0
  • 🤖 AI Added Allow: /africa/llms.txt$ for ChatGPT-User
  • 🤖 AI Added Allow: /africa/llms.txt/$ for ChatGPT-User
  • 🤖 AI Added Allow: /asia/llms.txt$ for ChatGPT-User
  • 🤖 AI Added Allow: /asia/llms.txt/$ for ChatGPT-User
  • 🤖 AI Added Allow: /canada/llms.txt$ for ChatGPT-User
  • 🤖 AI Added Allow: /canada/llms.txt/$ for ChatGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +3 −0
  • 🤖 AI Added Allow: /de-de/blog/what-it-takes-to-build-great-products-now/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/what-it-takes-to-build-great-products-now/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/what-it-takes-to-build-great-products-now/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/what-it-takes-to-build-great-products-now/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /es-es/blog/what-it-takes-to-build-great-products-now/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /fr-fr/blog/what-it-takes-to-build-great-products-now/$ for Claude-SearchBot
  • … 9 more change(s)
view diff → snapshot
SoundCloud soundcloud.com modified +29 −60
  • Blocked user-agent ‘Brightbot’ entirely
  • 🤖 AI Blocked omgili (Webz.io) entirely
  • Blocked user-agent ‘omgilibot’ entirely
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • … 45 more change(s)
view diff → snapshot
The Telegraph telegraph.co.uk modified +71 −205
  • Added Disallow: /application/* for *
  • Added Disallow: /etc.clientlibs/settings/wcm/designs/telegraph/core/clientlibs/page-refresh.*.js for *
  • Added Disallow: /etc.clientlibs/settings/wcm/designs/telegraph/core/clientlibs/save-article.*.js for *
  • Added Disallow: /ixale/ for *
  • Removed Disallow: */application/* for *
  • Removed Disallow: */ixale/ for *
  • … 45 more change(s)
view diff → snapshot
SoundCloud soundcloud.com modified +60 −29
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Blocked cohere-ai (Cohere) entirely
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +4 −0
  • 🤖 AI Added Allow: /resource-library/ai-design-assistant-tools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/best-ai-video-generators/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/how-to-make-a-cinemagraph/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/sizzle-reel/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/ai-design-assistant-tools/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /resource-library/best-ai-video-generators/$ for Claude-SearchBot
  • … 14 more change(s)
view diff → snapshot