NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2530snapshots
1357changes detected
230this week
All changes 🤖 AI-crawler changes Hiding initial archives
Figma Community figma.com modified +5 −0
  • 🤖 AI Added Allow: /de-de/loesungen/bild-vektorisieren/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-bildbearbeitung/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/solutions/changer-couleur-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/solutions/retouche-image-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/solutions/vectoriser-image/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/bild-vektorisieren/$ for Claude-SearchBot
  • … 14 more change(s)
view diff → snapshot
Zscaler zscaler.com modified +24 −1
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler cohere-ai (Cohere)
  • … 45 more change(s)
view diff → snapshot
Commonwealth Bank commbank.com.au modified +89 −104
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • … 45 more change(s)
view diff → snapshot
Weibo weibo.com modified +35 −15
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘ChatGLM’ entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • Blocked user-agent ‘DeepSeek’ entirely
  • Blocked user-agent ‘DeepSeek Chat’ entirely
  • Blocked user-agent ‘DeepSeekBot’ entirely
  • … 16 more change(s)
view diff → snapshot
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /blog/code-craft-and-the-making-of-nested-folders/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/ai-creative-process/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/ai-tools-for-graphic-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/claude-skill-design-system/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/design-operations/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/user-centered-design-questions/$ for CHATGPT-User
  • … 22 more change(s)
view diff → snapshot
Tanium tanium.com modified +24 −1
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Google-Extended (Google)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
  • … 4 more change(s)
view diff → snapshot
Facebook facebook.com modified +329 −21
  • 🤖 AI Unblocked Amazonbot (Amazon)
  • 🤖 AI Added Allow: /ajax/bootloader-endpoint/ for Amazonbot
  • 🤖 AI Added Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet for Amazonbot
  • 🤖 AI Added Allow: /careers/ for Amazonbot
  • 🤖 AI Added Allow: /safetycheck/ for Amazonbot
  • 🤖 AI Added Disallow: /*/plugins/* for Amazonbot
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for PerplexityBot
view diff → snapshot
Morgan Stanley morganstanley.com modified +46 −6
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • … 8 more change(s)
view diff → snapshot
Figma Community figma.com modified +24 −0
  • 🤖 AI Added Allow: /de-de/blog/how-decagon-uses-ai-for-design-system-saturation/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/sightlines-issue-no1/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/steal-this-template-bring-a-user-persona-to-life-with-figma-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/workflow-lab-deploying-designs-directly-with-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-decagon-uses-ai-for-design-system-saturation/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/sightlines-issue-no1/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +17 −0
  • 🤖 AI Added Allow: /blog/properties-panel-and-annotations-now-in-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-annotation-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-background-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-card-layout-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-color-palette-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-empty-state-generator/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Claude claude.ai modified +1 −1
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • Removed Disallow: /share/* for *
view diff → snapshot
Figma Community figma.com modified +3 −0
  • 🤖 AI Added Allow: /solutions/ai-frame-namer/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-page-organizer/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-variable-namer/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-frame-namer/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /solutions/ai-page-organizer/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /solutions/ai-variable-namer/$ for Claude-SearchBot
  • … 6 more change(s)
view diff → snapshot
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/customers/duotone-saves-hours/$ for CHATGPT-User
  • … 22 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /pt-br/customers/duotone-saves-hours/$ for PerplexityBot
view diff → snapshot
BBC bbc.com modified +7 −1
  • 🤖 AI Added Allow: /storyworks for ChatGPT-User
  • 🤖 AI Added Allow: /storyworks/ for ChatGPT-User
  • 🤖 AI Added Allow: /storyworks for Google-Extended
  • 🤖 AI Added Allow: /storyworks/ for Google-Extended
  • 🤖 AI Added Allow: /storyworks for GPTBot
  • 🤖 AI Added Allow: /storyworks/ for GPTBot
view diff → snapshot
Lithia Motors lithiamotors.com modified +61 −1
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • … 5 more change(s)
view diff → snapshot
Fortra fortra.com modified +1 −1
  • 🤖 AI Unblocked Google-Extended (Google)
  • 🤖 AI Added Allow: / for Google-Extended
view diff → snapshot
Figma Community figma.com modified +105 −1
  • 🤖 AI Added Allow: /blog/the-visual-identity-behind-config-2026/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/7-questions-from-lc/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/design-systems-104-making-metrics-matter/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/in-good-company-how-publishers-use-figma-to-help-design-the-news/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-ebenen-namensgeber/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/resource-library/ki-design-tools/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +0 −93
  • 🤖 AI Removed Allow: /blog/the-visual-identity-behind-config-2026/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/blog/7-questions-from-lc/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/blog/design-systems-104-making-metrics-matter/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/blog/in-good-company-how-publishers-use-figma-to-help-design-the-news/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/loesungen/ki-ebenen-namensgeber/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /es-es/blog/7-questions-from-lc/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +2 −2
  • 🤖 AI Added Allow: /blog/the-visual-identity-behind-config-2026/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-ebenen-namensgeber/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /customers/how-carrefour-leveraged-ai-to-drive-design-delivery/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /fr-fr/customers/how-carrefour-leveraged-ai-to-drive-design-delivery/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/the-visual-identity-behind-config-2026/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /de-de/loesungen/ki-ebenen-namensgeber/$ for Claude-SearchBot
  • … 10 more change(s)
view diff → snapshot
NBC News nbcnews.com modified +110 −0
  • Blocked user-agent ‘AhrefsBot’ entirely
  • Blocked user-agent ‘Archive.org_bot’ entirely
  • Blocked user-agent ‘Brightbot 1.0’ entirely
  • Blocked user-agent ‘ByteSpider Desktop’ entirely
  • Blocked user-agent ‘ExaBot’ entirely
  • Blocked user-agent ‘Petalbot’ entirely
  • … 5 more change(s)
view diff → snapshot
Figma Community figma.com modified +26 −0
  • 🤖 AI Added Allow: /it-it/soluzioni/assistente-handoff-design-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/soluzioni/controllo-accessibilita-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/soluzioni/convertitore-da-figma-a-codice/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/soluzioni/convertitore-da-mockup-a-prototipo/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/soluzioni/convertitore-da-sketch-a-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/soluzioni/denominazione-livelli-ai/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +67 −0
  • 🤖 AI Added Allow: /customers/how-carrefour-leveraged-ai-to-drive-design-delivery/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/7-questions-from-lc/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/design-systems-104-making-metrics-matter/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/in-good-company-how-publishers-use-figma-to-help-design-the-news/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/7-questions-from-lc/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/design-systems-104-making-metrics-matter/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/how-figma-stays-ahead-of-vulnerabilities-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/how-figma-stays-ahead-of-vulnerabilities-with-agents/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/how-figma-stays-ahead-of-vulnerabilities-with-agents/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/how-figma-stays-ahead-of-vulnerabilities-with-agents/$ for PerplexityBot
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/loesungen/ki-assistent-design-uebergabe/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-preistabellen-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/asistente-de-entrega-de-disenos-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/convertidor-de-figma-a-codigo/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/convertidor-de-mockup-a-prototipo/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/convertidor-de-sketch-a-figma/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot