NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1204files archived
3937snapshots
2782changes detected
216this week
All changes 🤖 AI-crawler changes Hiding initial archives
Academia.edu academia.edu modified +0 −2
  • 🤖 AI Removed Allow: /3 for Google-Extended
  • 🤖 AI Removed Allow: /9 for Google-Extended
view diff → snapshot
Indeed indeed.com modified +33 −17
  • Removed all rules for user-agent ‘Bingbot’
  • Added Disallow: /Zeige-Job for *
  • Added Disallow: /generate-question for *
  • Added Disallow: /hire/ats-partners for *
  • Added Disallow: /hire/o/ats-partners for *
  • Added Disallow: /m/viewjob for *
  • … 45 more change(s)
view diff → snapshot
Commonwealth Bank commbank.com.au modified +4 −1
  • Added Allow: /etc/designs/commbank/favicon.ico for *
  • 🤖 AI Added Allow: /digital/home-loans/borrowing-power-calculator for Amazonbot
  • 🤖 AI Added Allow: /etc/designs/commbank/favicon.ico for Amazonbot
  • 🤖 AI Added Allow: /digital/home-loans/borrowing-power-calculator for Applebot-Extended
  • 🤖 AI Added Allow: /etc/designs/commbank/favicon.ico for Applebot-Extended
  • 🤖 AI Added Allow: /digital/home-loans/borrowing-power-calculator for Bytespider
  • … 29 more change(s)
view diff → snapshot
GoDaddy godaddy.com modified +32 −11
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Perplexity-User (Perplexity)
  • 🤖 AI Added Allow: */domain-auctions/ for Claude-SearchBot
  • 🤖 AI Added Allow: */domainsearch/ for Claude-SearchBot
  • 🤖 AI Added Disallow: */domain-auctions/api/* for Claude-SearchBot
  • 🤖 AI Removed Allow: /domain-auctions/ for Claude-SearchBot
  • … 26 more change(s)
view diff → snapshot
Figma Community figma.com modified +14 −6
  • 🤖 AI Added Allow: /blog/show-your-work/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/source-material-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/workflow-lab-moving-between-design-and-code-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/workflow-lab-moving-between-design-and-code-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/workflow-lab-moving-between-design-and-code-with-agents/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
The Atlantic theatlantic.com modified +5 −1
  • Unblocked user-agent ‘Applebot’
  • Added Allow: / for Applebot
  • 🤖 AI Added Allow: /sponsored/ for Claude-SearchBot
  • 🤖 AI Added Allow: /sponsored/ for Claude-User
  • 🤖 AI Added Allow: /sponsored/ for Perplexity-User
  • 🤖 AI Added Allow: /sponsored/ for PerplexityBot
view diff → snapshot
Best Buy bestbuy.com modified +0 −11
  • Removed Disallow: */site/searchpage.jsp?st=* for *
  • Removed Disallow: *?cp=*&id=*&st=* for *
  • Removed Disallow: *?id*_facet*&st=* for *
  • Removed Disallow: *?id*_facet*_facet*&st=* for *
  • 🤖 AI Removed Disallow: *?cp=*&id=*&st=* for AmazonBot
  • 🤖 AI Removed Disallow: *?id*_facet*&st=* for AmazonBot
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/try-these-5-weave-tools-and-share-your-own/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/intro-to-weave-workflow/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/try-these-5-weave-tools-and-share-your-own/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /webinars/intro-to-weave-workflow/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/try-these-5-weave-tools-and-share-your-own/$ for Claude-User
  • 🤖 AI Added Allow: /webinars/intro-to-weave-workflow/$ for Claude-User
  • … 4 more change(s)
view diff → snapshot
GoDaddy godaddy.com modified +23 −0
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
  • 🤖 AI Added Allow: /domain-auctions/ for Claude-User
view diff → snapshot
Weather weather.com modified +20 −133
  • Added rules for user-agent ‘Brightbot 1.0’
  • Blocked user-agent ‘Google-CloudVertexBot’ entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
  • 🤖 AI Unblocked anthropic-ai (Anthropic) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Claude-Web (Anthropic) - rules removed
  • … 45 more change(s)
view diff → snapshot
Sony Music sonymusic.com modified +7 −60
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 6 more change(s)
view diff → snapshot
Expeditors International expeditors.com modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Tenet Healthcare tenethealth.com modified +61 −53
  • Added rules for user-agent ‘AdsBot-Google’
  • Added rules for user-agent ‘AdsBot-Google-Mobile’
  • Added rules for user-agent ‘AdsBot-Google-Mobile-Apps’
  • Added rules for user-agent ‘bingbot’
  • Added rules for user-agent ‘DuckDuckBot’
  • Added rules for user-agent ‘facebookexternalhit’
  • … 22 more change(s)
view diff → snapshot
Guyana (Office of the President) op.gov.gy modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Costa Rica (Presidencia) presidencia.go.cr modified +70 −59
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/source-material-zach-lieberman/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-zach-lieberman/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-zach-lieberman/$ for Claude-User
  • 🤖 AI Added Allow: /blog/source-material-zach-lieberman/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-zach-lieberman/$ for PerplexityBot
view diff → snapshot
EPFL epfl.ch modified +0 −61
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • … 4 more change(s)
view diff → snapshot
AI21 Labs ai21.com modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Iceland (Government.is) government.is modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +7 −6
  • 🤖 AI Added Allow: /de-de/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /es-la/blog/source-material-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-thom-bettridge/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /es-la/blog/source-material-thom-bettridge/$ for Claude-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-thom-bettridge/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /es-la/blog/source-material-thom-bettridge/$ for PerplexityBot
view diff → snapshot
Reuters reuters.com modified +0 −1
  • Removed Disallow: /jp/ for AASA-Bot
  • Removed Disallow: /jp/ for ADmantX
  • Removed Disallow: /jp/ for AdmiralBot
  • Removed Disallow: /jp/ for AdsBot-Google
  • Removed Disallow: /jp/ for AdsBot-Google-Mobile
  • Removed Disallow: /jp/ for AmazonAdBot
  • … 45 more change(s)
view diff → snapshot
Lululemon Athletica lululemon.com modified +9 −6
  • 🤖 AI Blocked Meta-WebIndexer (Meta) entirely
  • Added Allow: /*/c/*/*?*icid=* for *
  • Added Allow: /c/*/*?*icid=* for *
  • Added Disallow: /*/c/*/*z*z*z* for *
  • Added Disallow: /*/search?Ntt= for *
  • Added Disallow: /c/*/*z*z*z* for *
  • … 4 more change(s)
view diff → snapshot
xAI x.ai modified +2 −13
  • 🤖 AI Removed rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Removed rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Removed rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Removed rules for AI crawler Google-Extended (Google)
  • 🤖 AI Removed rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Removed rules for AI crawler PerplexityBot (Perplexity)
view diff → snapshot
Figma Community figma.com modified +25 −0
  • 🤖 AI Added Allow: /careers/early-career/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/claude-code-for-designers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/claude-skills-for-product-designers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/enterprise-design-systems/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/figma-mcp-use-cases/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/llm-context-design/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
El Corte Ingles elcorteingles.es modified +23 −20
  • Added rules for user-agent ‘AdsBot-Google-Mobile’
  • 🤖 AI Unblocked Claude-User (Anthropic) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Added Disallow: /*?*&analytics_data= for *
  • Added Disallow: /*?*&pdp_ctype= for *
  • Added Disallow: /*?*&term= for *
  • … 45 more change(s)
view diff → snapshot
GitHub github.com modified +140 −9
  • Added rules for user-agent ‘AdsBot-Google’
  • Added rules for user-agent ‘AdsBot-Google-Mobile’
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • … 2 more change(s)
view diff → snapshot
Zscaler zscaler.com modified +1 −1
  • 🤖 AI Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for Applebot-Extended
  • 🤖 AI Removed Disallow: /indexNowKey.txt for Applebot-Extended
  • 🤖 AI Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for ChatGPT-User
  • 🤖 AI Removed Disallow: /indexNowKey.txt for ChatGPT-User
  • 🤖 AI Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for Claude-SearchBot
  • 🤖 AI Removed Disallow: /indexNowKey.txt for Claude-SearchBot
  • … 14 more change(s)
view diff → snapshot
Figma Community figma.com modified +5 −2
  • 🤖 AI Added Allow: /blog/japan-local-data-hosting/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /id-id/resource-library/$ for CHATGPT-User
  • 🤖 AI Added Allow: /th-th/resource-library/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /blog/source-material-creative-technologist-marisa-chentakul-meshtimes/$ for CHATGPT-User
  • … 29 more change(s)
view diff → snapshot
Canva canva.com modified +3 −0
  • 🤖 AI Added Allow: /*/ai-connector/ for anthropic-ai
  • 🤖 AI Added Allow: /ai-connector/ for anthropic-ai
  • 🤖 AI Added Allow: /developers/ for anthropic-ai
  • 🤖 AI Added Allow: /*/ai-connector/ for Applebot-Extended
  • 🤖 AI Added Allow: /ai-connector/ for Applebot-Extended
  • 🤖 AI Added Allow: /developers/ for Applebot-Extended
  • … 12 more change(s)
view diff → snapshot