NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Showing initial archives
KeyCorp key.com modified +128 −5
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • … 13 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/try-these-10-skills-and-show-off-your-own/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/try-these-10-skills-and-show-off-your-own/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/try-these-10-skills-and-show-off-your-own/$ for Claude-User
  • 🤖 AI Added Allow: /blog/try-these-10-skills-and-show-off-your-own/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/try-these-10-skills-and-show-off-your-own/$ for PerplexityBot
view diff → snapshot
X (Twitter) x.com modified +18 −35
  • Added rules for user-agent ‘meta-externalads’
  • 🤖 AI Added rules for AI crawler meta-externalagent (Meta)
  • 🤖 AI Added rules for AI crawler meta-externalfetcher (Meta)
  • 🤖 AI Added rules for AI crawler Meta-WebIndexer (Meta)
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/3d-scene-generator-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-brand-video-generator/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +3 −0
  • 🤖 AI Added Allow: /es-la/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/get-started-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/how-we-secure-figmas-internal-systems-with-agents/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /pt-br/blog/how-we-secure-figmas-internal-systems-with-agents/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /webinars/get-started-with-the-figma-agent/$ for Claude-SearchBot
  • … 9 more change(s)
view diff → snapshot
Nvidia nvidia.com modified +18 −69
  • 🤖 AI Removed rules for AI crawler AI2Bot (Allen Institute for AI)
  • Removed all rules for user-agent ‘Amazon-kendra’
  • 🤖 AI Removed rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Removed rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Removed rules for AI crawler Applebot-Extended (Apple)
  • Removed all rules for user-agent ‘AutoGPT’
  • … 45 more change(s)
view diff → snapshot
Ecolab ecolab.com modified +29 −1
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
  • Added sitemap https://ecolab-dotcom-prod.vercel.app/sitemap-llm.xml
  • … 2 more change(s)
view diff → snapshot
Figma Community figma.com modified +30 −0
  • 🤖 AI Added Allow: /blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/code-craft-and-the-making-of-nested-folders/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/designer-framework-for-better-ai-prompts/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/how-figma-stays-ahead-of-vulnerabilities-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/code-craft-and-the-making-of-nested-folders/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/designer-framework-for-better-ai-prompts/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
USA Today usatoday.com modified +6 −0
  • Added Allow: /story/shopping/ for ChatGPT Operator
  • 🤖 AI Added Allow: /story/shopping/ for ChatGPT-User
  • 🤖 AI Added Allow: /story/shopping/ for Google-Extended
  • 🤖 AI Added Allow: /story/shopping/ for GPTBot
  • 🤖 AI Added Allow: /story/shopping/ for OAI-SearchBot
  • 🤖 AI Added Allow: /story/shopping/ for PerplexityBot
view diff → snapshot
Der Spiegel spiegel.de modified +209 −86
  • Blocked user-agent ‘AhrefsSiteAudit’ entirely
  • 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • Blocked user-agent ‘audisto’ entirely
  • Blocked user-agent ‘audisto-essential’ entirely
  • Blocked user-agent ‘barkrowler’ entirely
  • … 45 more change(s)
view diff → snapshot
Roblox roblox.com modified +61 −0
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • Blocked user-agent ‘CloudflareBrowserRenderingCrawler’ entirely
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +29 −0
  • 🤖 AI Added Allow: /de-de/blog/ai-fluency-isnt-the-finish-line/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/properties-panel-and-annotations-now-in-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/the-visual-identity-behind-config-2026/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/ai-fluency-isnt-the-finish-line/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/properties-panel-and-annotations-now-in-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/the-visual-identity-behind-config-2026/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +2 −1
  • 🤖 AI Blocked Claude-User (Anthropic) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Unblocked anthropic-ai (Anthropic) - rules removed
view diff → snapshot
Vimeo vimeo.com modified +45 −21
  • 🤖 AI Added Allow: /de/product/ for anthropic-ai
  • 🤖 AI Added Allow: /de/video-uploader for anthropic-ai
  • 🤖 AI Added Allow: /es/product/ for anthropic-ai
  • 🤖 AI Added Allow: /es/video-uploader for anthropic-ai
  • 🤖 AI Added Allow: /fr/product/ for anthropic-ai
  • 🤖 AI Added Allow: /fr/video-uploader for anthropic-ai
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +36 −0
  • 🤖 AI Added Allow: /es-es/soluciones/agente-de-diseno-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/cambiador-de-estilo-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/editor-de-fotos-por-lotes/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/generador-de-flujo-de-pago/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/indicacion-para-editar/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/revision-de-diseno/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Vimeo vimeo.com modified +21 −45
  • 🤖 AI Added Allow: /de/upload/ for anthropic-ai
  • 🤖 AI Added Allow: /es/upload/ for anthropic-ai
  • 🤖 AI Added Allow: /fr/upload/ for anthropic-ai
  • 🤖 AI Added Allow: /jp/upload/ for anthropic-ai
  • 🤖 AI Added Allow: /ko/upload/ for anthropic-ai
  • 🤖 AI Added Allow: /pt-br/upload/ for anthropic-ai
  • … 45 more change(s)
view diff → snapshot
USA Today usatoday.com modified +6 −0
  • Added Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ for ChatGPT Operator
  • 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ for ChatGPT-User
  • 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ for Google-Extended
  • 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ for GPTBot
  • 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ for OAI-SearchBot
  • 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/04/16/sell-your-home-for-top-dollar-by-beating-back-to-school-deadlines/89482853007/ for PerplexityBot
view diff → snapshot
Vimeo vimeo.com modified +45 −21
  • 🤖 AI Added Allow: /de/product/ for anthropic-ai
  • 🤖 AI Added Allow: /de/video-uploader for anthropic-ai
  • 🤖 AI Added Allow: /es/product/ for anthropic-ai
  • 🤖 AI Added Allow: /es/video-uploader for anthropic-ai
  • 🤖 AI Added Allow: /fr/product/ for anthropic-ai
  • 🤖 AI Added Allow: /fr/video-uploader for anthropic-ai
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +5 −0
  • 🤖 AI Added Allow: /fr-fr/fonts/geometric/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/fonts/humanist/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/fonts/neo-grotesque/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/fonts/rounded/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/fonts/script/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/fonts/geometric/$ for Claude-SearchBot
  • … 14 more change(s)
view diff → snapshot
Shutterstock shutterstock.com modified +16 −8
  • Added Disallow: /*/search/personalized/ for *
  • Added Disallow: /*/search/similar/ for *
  • Added Disallow: /search/personalized/ for *
  • Added Disallow: /search/similar/ for *
  • Removed Disallow: /search/personalized/* for *
  • Removed Disallow: /search/similar/* for *
  • … 18 more change(s)
view diff → snapshot
Figma Community figma.com modified +69 −0
  • 🤖 AI Added Allow: /de-de/loesungen/checkout-flow-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/design-review/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-design-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-stilwechsler/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/prompt-zum-bearbeiten/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/stapelbildeditor/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Fortra fortra.com modified +2 −2
  • 🤖 AI Unblocked ClaudeBot (Anthropic)
  • 🤖 AI Added Allow: / for ClaudeBot
  • 🤖 AI Unblocked GPTBot (OpenAI)
  • 🤖 AI Added Allow: / for GPTBot
view diff → snapshot
The Economist economist.com modified +2 −0
  • 🤖 AI Added Allow: /pro$ for Google-Extended
  • 🤖 AI Added Allow: /pro/ for Google-Extended
view diff → snapshot
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /de-de/reports/lessons-from-leaders-designing-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/reports/lessons-from-leaders-designing-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/reports/lessons-from-leaders-designing-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/reports/lessons-from-leaders-designing-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/reports/lessons-from-leaders-designing-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/reports/lessons-from-leaders-designing-agents/$ for CHATGPT-User
  • … 22 more change(s)
view diff → snapshot
SAP sap.com modified +3 −1
  • Added Disallow: /*/silent.html for 360Spider
  • Added Disallow: /silent.html for 360Spider
  • 🤖 AI Added Disallow: /*/silent.html for Amazonbot
  • 🤖 AI Added Disallow: /silent.html for Amazonbot
  • Added Disallow: /*/silent.html for archive.org_bot
  • Added Disallow: /silent.html for archive.org_bot
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +21 −0
  • 🤖 AI Added Allow: /blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/what-matters-when-anyone-can-build/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/what-the-design-to-code-loop-unlocks/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/what-matters-when-anyone-can-build/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/what-the-design-to-code-loop-unlocks/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/what-matters-when-anyone-can-build/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Cencora cencora.com modified +37 −1
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • … 7 more change(s)
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/loesungen/farbe-aendern/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-design-assistent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/fonts/dm-sans/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/solutions/ai-change-color/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/solutions/ai-design-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/solutions/ai-design-assistant/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +5 −0
  • 🤖 AI Added Allow: /de-de/loesungen/bild-vektorisieren/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-bildbearbeitung/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/solutions/changer-couleur-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/solutions/retouche-image-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/solutions/vectoriser-image/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/bild-vektorisieren/$ for Claude-SearchBot
  • … 14 more change(s)
view diff → snapshot