NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2571snapshots
1398changes detected
225this week
All changes 🤖 AI-crawler changes Showing initial archives
Figma Community figma.com modified +26 −0
  • 🤖 AI Added Allow: /it-it/soluzioni/assistente-handoff-design-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/soluzioni/controllo-accessibilita-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/soluzioni/convertitore-da-figma-a-codice/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/soluzioni/convertitore-da-mockup-a-prototipo/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/soluzioni/convertitore-da-sketch-a-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/soluzioni/denominazione-livelli-ai/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +67 −0
  • 🤖 AI Added Allow: /customers/how-carrefour-leveraged-ai-to-drive-design-delivery/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/7-questions-from-lc/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/design-systems-104-making-metrics-matter/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/in-good-company-how-publishers-use-figma-to-help-design-the-news/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/7-questions-from-lc/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/design-systems-104-making-metrics-matter/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/how-figma-stays-ahead-of-vulnerabilities-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/how-figma-stays-ahead-of-vulnerabilities-with-agents/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/how-figma-stays-ahead-of-vulnerabilities-with-agents/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/how-figma-stays-ahead-of-vulnerabilities-with-agents/$ for PerplexityBot
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/loesungen/ki-assistent-design-uebergabe/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-preistabellen-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/asistente-de-entrega-de-disenos-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/convertidor-de-figma-a-codigo/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/convertidor-de-mockup-a-prototipo/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/convertidor-de-sketch-a-figma/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Reuters reuters.com modified +2 −1
  • Added Disallow: /account/subscribe/payment for AASA-Bot
  • Added Disallow: /jp/ for AASA-Bot
  • Removed Disallow: /account/subscribe/payment/ for AASA-Bot
  • Added Disallow: /account/subscribe/payment for ADmantX
  • Added Disallow: /jp/ for ADmantX
  • Removed Disallow: /account/subscribe/payment/ for ADmantX
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +17 −0
  • 🤖 AI Added Allow: /customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/4-ways-were-using-our-mcp-server-at-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/figma-zu-code-konverter/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/mockup-zu-prototyp-konverter/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/sketch-zu-figma-konverter/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/4-ways-were-using-our-mcp-server-at-figma/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
USA Today usatoday.com modified +6 −0
  • Added Allow: /money/money-management/ for ChatGPT Operator
  • 🤖 AI Added Allow: /money/money-management/ for ChatGPT-User
  • 🤖 AI Added Allow: /money/money-management/ for Google-Extended
  • 🤖 AI Added Allow: /money/money-management/ for GPTBot
  • 🤖 AI Added Allow: /money/money-management/ for OAI-SearchBot
  • 🤖 AI Added Allow: /money/money-management/ for PerplexityBot
view diff → snapshot
The Washington Post washingtonpost.com modified +3 −1
  • 🤖 AI Blocked PerplexityBot (Perplexity) entirely
  • 🤖 AI Added Allow: /advertising/ for PerplexityBot
  • 🤖 AI Added Allow: /creativegroup/ for PerplexityBot
  • 🤖 AI Removed Allow: / for PerplexityBot
view diff → snapshot
Figma Community figma.com modified +11 −0
  • 🤖 AI Added Allow: /de-de/loesungen/ki-barrierefreiheits-checker/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/solutions/verificateur-accessibilite-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/2026-ai-report/$ for CHATGPT-User
  • … 38 more change(s)
view diff → snapshot
Roblox roblox.com modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
SoundCloud soundcloud.com modified +29 −60
  • Blocked user-agent ‘Brightbot’ entirely
  • 🤖 AI Blocked omgili (Webz.io) entirely
  • Blocked user-agent ‘omgilibot’ entirely
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +31 −0
  • 🤖 AI Added Allow: /de-de/blog/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/jen-dunnam-lead-design-teams/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/sightlines/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-footer-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-hintergrundentferner/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Roblox roblox.com modified +61 −0
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • Blocked user-agent ‘CloudflareBrowserRenderingCrawler’ entirely
  • … 4 more change(s)
view diff → snapshot
The Washington Post washingtonpost.com modified +1 −3
  • 🤖 AI Unblocked PerplexityBot (Perplexity)
  • 🤖 AI Added Allow: / for PerplexityBot
  • 🤖 AI Removed Allow: /advertising/ for PerplexityBot
  • 🤖 AI Removed Allow: /creativegroup/ for PerplexityBot
view diff → snapshot
SoundCloud soundcloud.com modified +60 −29
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Blocked cohere-ai (Cohere) entirely
  • … 45 more change(s)
view diff → snapshot
Roblox roblox.com modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Waldo waldo.fyi modified +9 −0
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Google-Extended (Google)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • … 1 more change(s)
view diff → snapshot
Le Figaro lefigaro.fr modified +2 −0
  • 🤖 AI Added Allow: /beaute for ChatGPT-User
  • 🤖 AI Added Allow: /beaute for OAI-SearchBot
view diff → snapshot
The Economist economist.com modified +19 −1
  • 🤖 AI Blocked Claude-SearchBot (Anthropic) entirely
  • 🤖 AI Blocked Claude-User (Anthropic) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Added Allow: /pro$ for ChatGPT-User
  • 🤖 AI Added Allow: /pro$ for Claude-Web
  • 🤖 AI Added Allow: /pro$ for OAI-SearchBot
  • … 2 more change(s)
view diff → snapshot
Figma Community figma.com modified +22 −1
  • 🤖 AI Added Allow: /de-de/resource-library/vibe-coding-beispiele/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/resource-library/ejemplos-de-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/resource-library/ejemplos-de-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/resource-library/exemples-de-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /id-id/resource-library/contoh-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/resource-library/esempi-di-vibe-coding/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Roblox roblox.com modified +61 −0
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • Blocked user-agent ‘CloudflareBrowserRenderingCrawler’ entirely
  • … 4 more change(s)
view diff → snapshot
Weibo weibo.com modified +15 −35
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • Removed all rules for user-agent ‘ChatGLM’
  • 🤖 AI Unblocked ChatGPT-User (OpenAI) - rules removed
  • Removed all rules for user-agent ‘DeepSeek’
  • Removed all rules for user-agent ‘DeepSeek Chat’
  • Removed all rules for user-agent ‘DeepSeekBot’
  • … 16 more change(s)
view diff → snapshot
Weibo weibo.com modified +35 −15
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘ChatGLM’ entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • Blocked user-agent ‘DeepSeek’ entirely
  • Blocked user-agent ‘DeepSeek Chat’ entirely
  • Blocked user-agent ‘DeepSeekBot’ entirely
  • … 16 more change(s)
view diff → snapshot
Ford Motor ford.com modified +2 −46
  • Removed all rules for user-agent ‘amazon-kendra’
  • Removed all rules for user-agent ‘amazon-QBusiness’
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Removed rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Removed rules for AI crawler Applebot-Extended (Apple)
  • Removed all rules for user-agent ‘bedrockbot’
  • … 26 more change(s)
view diff → snapshot
Figma Community figma.com modified +15 −0
  • 🤖 AI Added Allow: /de-de/blog/improving-performance-in-the-layers-panel/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/improving-performance-in-the-layers-panel/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/improving-performance-in-the-layers-panel/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/improving-performance-in-the-layers-panel/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/improving-performance-in-the-layers-panel/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/improving-performance-in-the-layers-panel/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Snapchat snapchat.com modified +42 −23
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler Perplexity-User (Perplexity)
  • Removed all rules for user-agent ‘Perplexity‑User’
  • Removed Disallow: /entertainment/ for *
  • 🤖 AI Unblocked ChatGPT-User (OpenAI)
  • … 17 more change(s)
view diff → snapshot
The Economist economist.com modified +49 −21
  • 🤖 AI Blocked OAI-SearchBot (OpenAI) entirely
  • 🤖 AI Added Allow: /pro/ for ChatGPT-User
  • 🤖 AI Added Allow: /pro/ for Claude-Web
  • 🤖 AI Added Allow: /pro/ for Perplexity-User
  • 🤖 AI Added Allow: /pro/ for PerplexityBot
view diff → snapshot
Figma Community figma.com modified +43 −0
  • 🤖 AI Added Allow: /blog/sightlines-issue-no1/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/figma-make-general-availability/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/future-states/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/gpt-5-6-is-now-available-in-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/figma-make-general-availability/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/future-states/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Europe PMC europepmc.org modified +98 −7
  • Added rules for user-agent ‘baiduspider’
  • Added rules for user-agent ‘bingbot’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
  • … 21 more change(s)
view diff → snapshot