NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
Zscaler zscaler.com modified +0 −7
  • 🤖 AI Removed Disallow: /*?* for Applebot-Extended
  • 🤖 AI Removed Disallow: /*?* for ChatGPT-User
  • 🤖 AI Removed Disallow: /*?* for Claude-SearchBot
  • 🤖 AI Removed Disallow: /*?* for Claude-User
  • 🤖 AI Removed Disallow: /*?* for ClaudeBot
  • 🤖 AI Removed Disallow: /*?* for Cohere-ai
  • … 4 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +22 −0
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • 🤖 AI Blocked GPTBot (OpenAI) entirely
  • … 2 more change(s)
view diff → snapshot
Yandex yandex.ru modified +272 −2
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • Added Disallow: /finance/banks/*/reviews*? for *
  • Added Disallow: /support2/sport for *
  • Removed Disallow: /finance/banks/*/reviews*?* for *
  • Removed Disallow: /support2/sport* for *
view diff → snapshot
Bank Central Asia bca.co.id modified +1 −9
  • 🤖 AI Removed rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Removed rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Removed rules for AI crawler OAI-SearchBot (OpenAI)
  • Added Disallow: /id/Forms/Ext/event-bca for *
view diff → snapshot
Kohl's kohls.com modified +0 −6
  • 🤖 AI Unblocked meta-externalagent (Meta) - rules removed
  • 🤖 AI Unblocked Meta-WebIndexer (Meta) - rules removed
view diff → snapshot
Figma Community figma.com modified +5 −0
  • 🤖 AI Added Allow: /customers/how-carrefour-leveraged-ai-to-drive-design-delivery/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/source-material-zach-lieberman/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/customers/how-carrefour-leveraged-ai-to-drive-design-delivery/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/source-material-zach-lieberman/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/blog/source-material-zach-lieberman/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/how-carrefour-leveraged-ai-to-drive-design-delivery/$ for Claude-SearchBot
  • … 19 more change(s)
view diff → snapshot
Figma Community figma.com modified +23 −0
  • 🤖 AI Added Allow: /de-de/blog/source-material-zach-lieberman/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/try-these-5-weave-tools-and-share-your-own/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-dashboard-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-zach-lieberman/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/try-these-5-weave-tools-and-share-your-own/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/agente-de-paneles-de-ia/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
NPR npr.org modified +6 −0
  • 🤖 AI Blocked Claude-SearchBot (Anthropic) entirely
  • 🤖 AI Blocked Claude-User (Anthropic) entirely
view diff → snapshot
Bank Central Asia bca.co.id modified +9 −0
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
view diff → snapshot
Insight Enterprises insight.com modified +91 −19
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • 🤖 AI Blocked GPTBot (OpenAI) entirely
  • 🤖 AI Blocked meta-externalagent (Meta) entirely
  • … 37 more change(s)
view diff → snapshot
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /blog/source-material-aitana-pico/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/csv-to-chart-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/csv-to-dashboard-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/image-to-icon-set-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/logo-to-brand-kit-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/prd-to-prototype-generator/$ for CHATGPT-User
  • … 29 more change(s)
view diff → snapshot
Figma Community figma.com modified +9 −0
  • 🤖 AI Added Allow: /resource-library/ai-art-styles/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/ai-product-photography-examples/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/ai-storyboard-examples/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/contextual-ai-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/design-system-governance/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/design-token-architecture/$ for CHATGPT-User
  • … 39 more change(s)
view diff → snapshot
Baxter International baxter.com modified +2 −133
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /ja-jp/blog/japan-local-data-hosting/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/japan-local-data-hosting/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /ja-jp/blog/japan-local-data-hosting/$ for Claude-User
  • 🤖 AI Added Allow: /ja-jp/blog/japan-local-data-hosting/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /ja-jp/blog/japan-local-data-hosting/$ for PerplexityBot
view diff → snapshot
Academia.edu academia.edu modified +0 −2
  • 🤖 AI Removed Allow: /3 for Google-Extended
  • 🤖 AI Removed Allow: /9 for Google-Extended
view diff → snapshot
Indeed indeed.com modified +33 −17
  • Removed all rules for user-agent ‘Bingbot’
  • Added Disallow: /Zeige-Job for *
  • Added Disallow: /generate-question for *
  • Added Disallow: /hire/ats-partners for *
  • Added Disallow: /hire/o/ats-partners for *
  • Added Disallow: /m/viewjob for *
  • … 45 more change(s)
view diff → snapshot
Commonwealth Bank commbank.com.au modified +4 −1
  • Added Allow: /etc/designs/commbank/favicon.ico for *
  • 🤖 AI Added Allow: /digital/home-loans/borrowing-power-calculator for Amazonbot
  • 🤖 AI Added Allow: /etc/designs/commbank/favicon.ico for Amazonbot
  • 🤖 AI Added Allow: /digital/home-loans/borrowing-power-calculator for Applebot-Extended
  • 🤖 AI Added Allow: /etc/designs/commbank/favicon.ico for Applebot-Extended
  • 🤖 AI Added Allow: /digital/home-loans/borrowing-power-calculator for Bytespider
  • … 29 more change(s)
view diff → snapshot
GoDaddy godaddy.com modified +32 −11
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Perplexity-User (Perplexity)
  • 🤖 AI Added Allow: */domain-auctions/ for Claude-SearchBot
  • 🤖 AI Added Allow: */domainsearch/ for Claude-SearchBot
  • 🤖 AI Added Disallow: */domain-auctions/api/* for Claude-SearchBot
  • 🤖 AI Removed Allow: /domain-auctions/ for Claude-SearchBot
  • … 26 more change(s)
view diff → snapshot
Figma Community figma.com modified +14 −6
  • 🤖 AI Added Allow: /blog/show-your-work/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/source-material-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/workflow-lab-moving-between-design-and-code-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/workflow-lab-moving-between-design-and-code-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/workflow-lab-moving-between-design-and-code-with-agents/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
The Atlantic theatlantic.com modified +5 −1
  • Unblocked user-agent ‘Applebot’
  • Added Allow: / for Applebot
  • 🤖 AI Added Allow: /sponsored/ for Claude-SearchBot
  • 🤖 AI Added Allow: /sponsored/ for Claude-User
  • 🤖 AI Added Allow: /sponsored/ for Perplexity-User
  • 🤖 AI Added Allow: /sponsored/ for PerplexityBot
view diff → snapshot
Best Buy bestbuy.com modified +0 −11
  • Removed Disallow: */site/searchpage.jsp?st=* for *
  • Removed Disallow: *?cp=*&id=*&st=* for *
  • Removed Disallow: *?id*_facet*&st=* for *
  • Removed Disallow: *?id*_facet*_facet*&st=* for *
  • 🤖 AI Removed Disallow: *?cp=*&id=*&st=* for AmazonBot
  • 🤖 AI Removed Disallow: *?id*_facet*&st=* for AmazonBot
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/try-these-5-weave-tools-and-share-your-own/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/intro-to-weave-workflow/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/try-these-5-weave-tools-and-share-your-own/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /webinars/intro-to-weave-workflow/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/try-these-5-weave-tools-and-share-your-own/$ for Claude-User
  • 🤖 AI Added Allow: /webinars/intro-to-weave-workflow/$ for Claude-User
  • … 4 more change(s)
view diff → snapshot
GoDaddy godaddy.com modified +23 −0
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
  • 🤖 AI Added Allow: /domain-auctions/ for Claude-User
view diff → snapshot
Weather weather.com modified +20 −133
  • Added rules for user-agent ‘Brightbot 1.0’
  • Blocked user-agent ‘Google-CloudVertexBot’ entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
  • 🤖 AI Unblocked anthropic-ai (Anthropic) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Claude-Web (Anthropic) - rules removed
  • … 45 more change(s)
view diff → snapshot
Sony Music sonymusic.com modified +7 −60
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 6 more change(s)
view diff → snapshot
Expeditors International expeditors.com modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Tenet Healthcare tenethealth.com modified +61 −53
  • Added rules for user-agent ‘AdsBot-Google’
  • Added rules for user-agent ‘AdsBot-Google-Mobile’
  • Added rules for user-agent ‘AdsBot-Google-Mobile-Apps’
  • Added rules for user-agent ‘bingbot’
  • Added rules for user-agent ‘DuckDuckBot’
  • Added rules for user-agent ‘facebookexternalhit’
  • … 22 more change(s)
view diff → snapshot
Guyana (Office of the President) op.gov.gy modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Costa Rica (Presidencia) presidencia.go.cr modified +70 −59
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/source-material-zach-lieberman/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-zach-lieberman/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-zach-lieberman/$ for Claude-User
  • 🤖 AI Added Allow: /blog/source-material-zach-lieberman/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-zach-lieberman/$ for PerplexityBot
view diff → snapshot