NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2571snapshots
1398changes detected
225this week
All changes 🤖 AI-crawler changes Hiding initial archives
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/workflow-lab-deploying-designs-directly-with-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/design-explore/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/workflow-lab-deploying-designs-directly-with-figma-make/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /es-la/design-explore/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/workflow-lab-deploying-designs-directly-with-figma-make/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /es-la/design-explore/$ for OAI-SearchBot
  • … 2 more change(s)
view diff → snapshot
Ford Motor ford.com modified +46 −0
  • Blocked user-agent ‘amazon-kendra’ entirely
  • Blocked user-agent ‘amazon-QBusiness’ entirely
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Blocked user-agent ‘bedrockbot’ entirely
  • … 24 more change(s)
view diff → snapshot
Europe PMC europepmc.org modified +7 −98
  • Removed all rules for user-agent ‘baiduspider’
  • Removed all rules for user-agent ‘bingbot’
  • 🤖 AI Removed rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Removed rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Removed rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Removed rules for AI crawler PerplexityBot (Perplexity)
  • … 21 more change(s)
view diff → snapshot
Figma Community figma.com modified +10 −0
  • 🤖 AI Added Allow: /blog/figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/how-trackman-built-a-design-first-culture-and-a-competitive-edge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/7-tips-for-using-figma-make-credits-more-efficiently/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/7-tips-for-using-figma-make-credits-more-efficiently/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/7-tips-for-using-figma-make-credits-more-efficiently/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/7-tips-for-using-figma-make-credits-more-efficiently/$ for CHATGPT-User
  • … 34 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /customers/mitsubishi-electric-corporation/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/mitsubishi-electric-corporation/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /customers/mitsubishi-electric-corporation/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /customers/mitsubishi-electric-corporation/$ for PerplexityBot
view diff → snapshot
Tripadvisor tripadvisor.com modified +0 −1
  • Removed Disallow: /ReviewExpress- for *
  • Removed Disallow: /ReviewExpress- for applebot
  • Removed Disallow: /ReviewExpress- for bingbot
  • 🤖 AI Removed Disallow: /ReviewExpress- for ChatGPT-User
  • Removed Disallow: /ReviewExpress- for ChatGPT-User/2.0
  • Removed Disallow: /ReviewExpress- for Gemini-Deep-Research
  • … 3 more change(s)
view diff → snapshot
Figma Community figma.com modified +21 −0
  • 🤖 AI Added Allow: /customers/how-shakers-uses-figma-to-align-teams-accelerate-delivery-and-elevate-craft/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/building-accessibility-into-a-canvas-based-product/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/customers/how-insider-one-customers-convert-designs-into-campaigns-instantly-with-figma-mcp/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/building-accessibility-into-a-canvas-based-product/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/customers/how-shakers-uses-figma-to-align-teams-accelerate-delivery-and-elevate-craft/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /es-la/customers/how-insider-one-customers-convert-designs-into-campaigns-instantly-with-figma-mcp/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/customers/how-insider-one-customers-convert-designs-into-campaigns-instantly-with-figma-mcp/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /es-la/customers/how-insider-one-customers-convert-designs-into-campaigns-instantly-with-figma-mcp/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /es-la/customers/how-insider-one-customers-convert-designs-into-campaigns-instantly-with-figma-mcp/$ for PerplexityBot
view diff → snapshot
Hulu hulu.com modified +0 −1
  • 🤖 AI Removed Disallow: /start for Amazonbot
  • 🤖 AI Removed Disallow: /start for Applebot-Extended
  • 🤖 AI Removed Disallow: /start for CCBot
  • 🤖 AI Removed Disallow: /start for ChatGPT-User
  • 🤖 AI Removed Disallow: /start for Claude-SearchBot
  • 🤖 AI Removed Disallow: /start for Claude-User
  • … 12 more change(s)
view diff → snapshot
Figma Community figma.com modified +28 −0
  • 🤖 AI Added Allow: /blog/how-decagon-uses-ai-for-design-system-saturation/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/figma-is-now-iso-42001-certified/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/customers/how-lollypop-takes-clients-from-brief-to-build-in-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/figma-is-now-iso-42001-certified/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/the-figma-agent-is-here/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /ja-jp/customers/lineyahoo/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/customers/lineyahoo/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /ja-jp/customers/lineyahoo/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /ja-jp/customers/lineyahoo/$ for PerplexityBot
view diff → snapshot
Figma Community figma.com modified +3 −1
  • 🤖 AI Added Allow: /blog/gpt-5-6-is-now-available-in-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/jen-dunnam-lead-design-teams/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/sightlines/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /blog/jen-dunham-lead-design-teams/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/gpt-5-6-is-now-available-in-figma-make/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/jen-dunnam-lead-design-teams/$ for Claude-SearchBot
  • … 10 more change(s)
view diff → snapshot
Figma Community figma.com modified +4 −0
  • 🤖 AI Added Allow: /blog/jen-dunham-lead-design-teams/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/botim-ai-ready-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/customers/botim-ai-ready-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/claude-skills-for-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/jen-dunham-lead-design-teams/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /customers/botim-ai-ready-design/$ for Claude-SearchBot
  • … 10 more change(s)
view diff → snapshot
Huntington Ingalls Industries hii.com modified +70 −3
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Added rules for user-agent ‘bingbot’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler cohere-ai (Cohere)
  • … 9 more change(s)
view diff → snapshot
Slate slate.com modified +20 −0
  • Added rules for user-agent ‘AdmixerBot’
  • Added rules for user-agent ‘AffsignalCrawler’
  • 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
  • Blocked user-agent ‘CohereBot’ entirely
  • Added rules for user-agent ‘Dragonfly’
  • Added rules for user-agent ‘Google-adstxt’
  • … 14 more change(s)
view diff → snapshot
Bloomberg bloomberg.com modified +48 −7
  • 🤖 AI Added Allow: /company for Claude-SearchBot
  • 🤖 AI Added Allow: /faq for Claude-SearchBot
  • 🤖 AI Added Allow: /latam for Claude-SearchBot
  • 🤖 AI Added Allow: /professional for Claude-SearchBot
  • 🤖 AI Added Allow: /tc for Claude-SearchBot
  • 🤖 AI Added Allow: /company for Claude-User
  • … 41 more change(s)
view diff → snapshot
Trend Micro trendmicro.com modified +3 −10
  • Removed all rules for user-agent ‘ChatGPT-User/2.0’
  • 🤖 AI Removed Allow: *.css$ for Amazonbot
  • 🤖 AI Removed Allow: *.js$ for Amazonbot
  • 🤖 AI Removed Disallow: /*/business/parsing-pages/ for Amazonbot
  • 🤖 AI Removed Disallow: /*/business/trials_parsing_pages/ for Amazonbot
  • 🤖 AI Removed Disallow: /download/* for Amazonbot
  • … 45 more change(s)
view diff → snapshot
Hulu hulu.com modified +132 −207
  • Added rules for user-agent ‘360Spider’
  • Added rules for user-agent ‘Adidxbot’
  • Added rules for user-agent ‘AdsBot-Google’
  • Added rules for user-agent ‘AdsBot-Google-Mobile’
  • Added rules for user-agent ‘AdsBot-Google-Mobile-Apps’
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +15 −0
  • 🤖 AI Added Allow: /de-de/blog/pantone-speaking-the-language-of-color/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/principles-in-motion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/pantone-speaking-the-language-of-color/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/principles-in-motion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/pantone-speaking-the-language-of-color/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/principles-in-motion/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
SoundCloud soundcloud.com modified +29 −60
  • Blocked user-agent ‘Brightbot’ entirely
  • 🤖 AI Blocked omgili (Webz.io) entirely
  • Blocked user-agent ‘omgilibot’ entirely
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • … 45 more change(s)
view diff → snapshot
Palo Alto Networks paloaltonetworks.com modified +108 −0
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
view diff → snapshot
SoundCloud soundcloud.com modified +60 −29
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Blocked cohere-ai (Cohere) entirely
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /es-la/blog/in-good-company-retailers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/in-good-company-retailers/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /es-la/blog/in-good-company-retailers/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /es-la/blog/in-good-company-retailers/$ for PerplexityBot
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/7-questions-from-lc/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/build-with-joy/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/7-questions-from-lc/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/build-with-joy/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/7-questions-from-lc/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/build-with-joy/$ for OAI-SearchBot
  • … 2 more change(s)
view diff → snapshot
Figma Community figma.com modified +68 −0
  • 🤖 AI Added Allow: /de-de/blog/introducing-keyboard-accessibility-features/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/new-features-in-figma-buzz/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/4-new-ways-to-go-from-idea-to-product-with-ai-tools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/5-shifts-redefining-design-systems-in-the-ai-era/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/figma-make-now-on-your-local-code/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/you-never-stop-cultivating-taste/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Kuehne+Nagel kuehne-nagel.com modified +33 −1
  • Added rules for user-agent ‘amazon-Qbusiness’
  • Added rules for user-agent ‘Applebot’
  • Added rules for user-agent ‘Baiduspider’
  • 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • Added rules for user-agent ‘SERankingBacklinksBot’
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for PerplexityBot
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/building-accessibility-into-a-canvas-based-product/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/building-accessibility-into-a-canvas-based-product/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/building-accessibility-into-a-canvas-based-product/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for OAI-SearchBot
  • … 2 more change(s)
view diff → snapshot
Volvo Cars volvocars.com modified +5 −2
  • Added rules for user-agent ‘YisouSpider’
  • 🤖 AI Added Disallow: /*/build/* for Amazonbot
  • 🤖 AI Added Disallow: /*/order-request/* for Amazonbot
  • 🤖 AI Removed Disallow: /*/build/*?* for Amazonbot
  • 🤖 AI Removed Disallow: /*/order* for Amazonbot
  • 🤖 AI Added Disallow: /*/build/* for anthropic-ai
  • … 45 more change(s)
view diff → snapshot