NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
LG Corp lg.com modified +123 −192
  • Removed all rules for user-agent ‘Baiduspider’
  • Removed all rules for user-agent ‘BingPreview’
  • 🤖 AI Removed rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Removed rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Removed rules for AI crawler Google-Extended (Google)
  • Removed all rules for user-agent ‘Googlebot’
  • … 22 more change(s)
view diff → snapshot
Intel intel.com modified +23 −1
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Google-Extended (Google)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
  • … 2 more change(s)
view diff → snapshot
Europe PMC europepmc.org modified +0 −141
  • Removed all rules for user-agent ‘baiduspider’
  • Removed all rules for user-agent ‘bingbot’
  • 🤖 AI Removed rules for AI crawler ChatGPT-User (OpenAI)
  • Removed all rules for user-agent ‘Googlebot’
  • Removed all rules for user-agent ‘MSNbot’
  • 🤖 AI Removed rules for AI crawler OAI-SearchBot (OpenAI)
  • … 19 more change(s)
view diff → snapshot
Europe PMC europepmc.org modified +141 −0
  • Added rules for user-agent ‘baiduspider’
  • Added rules for user-agent ‘bingbot’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • Added rules for user-agent ‘Googlebot’
  • Added rules for user-agent ‘MSNbot’
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • … 18 more change(s)
view diff → snapshot
DBLP dblp.org modified +95 −66
  • Blocked user-agent ‘AddSearchBot’ entirely
  • 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
  • Blocked user-agent ‘Ai2Bot-Dolma’ entirely
  • Blocked user-agent ‘aiHitBot’ entirely
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • Blocked user-agent ‘Andibot’ entirely
  • … 45 more change(s)
view diff → snapshot
USA Today usatoday.com modified +6 −12
  • Added Allow: /branded-story/ for ChatGPT Operator
  • Removed Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT Operator
  • Removed Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT Operator
  • 🤖 AI Added Allow: /branded-story/ for ChatGPT-User
  • 🤖 AI Removed Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT-User
  • 🤖 AI Removed Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT-User
  • … 12 more change(s)
view diff → snapshot
USA Today usatoday.com modified +12 −6
  • Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT Operator
  • Added Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT Operator
  • Removed Allow: /branded-story/ for ChatGPT Operator
  • 🤖 AI Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT-User
  • 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT-User
  • 🤖 AI Removed Allow: /branded-story/ for ChatGPT-User
  • … 12 more change(s)
view diff → snapshot
USA Today usatoday.com modified +6 −12
  • Added Allow: /branded-story/ for ChatGPT Operator
  • Removed Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT Operator
  • Removed Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT Operator
  • 🤖 AI Added Allow: /branded-story/ for ChatGPT-User
  • 🤖 AI Removed Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT-User
  • 🤖 AI Removed Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT-User
  • … 12 more change(s)
view diff → snapshot
Canva canva.com modified +2 −0
  • 🤖 AI Added Allow: /*/design-school/ for anthropic-ai
  • 🤖 AI Added Allow: /design-school/ for anthropic-ai
  • 🤖 AI Added Allow: /*/design-school/ for Applebot-Extended
  • 🤖 AI Added Allow: /design-school/ for Applebot-Extended
  • 🤖 AI Added Allow: /*/design-school/ for CCBot
  • 🤖 AI Added Allow: /design-school/ for CCBot
  • … 6 more change(s)
view diff → snapshot
Best Buy bestbuy.com modified +0 −1
  • 🤖 AI Removed Disallow: */site/searchpage.jsp?st=* for AmazonBot
  • Removed Disallow: */site/searchpage.jsp?st=* for Applebot
  • 🤖 AI Removed Disallow: */site/searchpage.jsp?st=* for ChatGPT-User
  • 🤖 AI Removed Disallow: */site/searchpage.jsp?st=* for Claude-User
  • 🤖 AI Removed Disallow: */site/searchpage.jsp?st=* for Claudebot
  • Removed Disallow: */site/searchpage.jsp?st=* for Gemeni-AI
  • … 6 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /es-la/blog/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/start-anywhere/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /es-la/blog/start-anywhere/$ for Claude-User
  • 🤖 AI Added Allow: /es-la/blog/start-anywhere/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /es-la/blog/start-anywhere/$ for PerplexityBot
view diff → snapshot
Figma Community figma.com modified +4 −0
  • 🤖 AI Added Allow: /webinars/building-your-first-component-with-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/connecting-reviewing-and-shipping-your-component/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/designer-to-builder-series/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/getting-ready-to-build-from-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/building-your-first-component-with-ai/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /webinars/connecting-reviewing-and-shipping-your-component/$ for Claude-SearchBot
  • … 14 more change(s)
view diff → snapshot
Figma Community figma.com modified +13 −0
  • 🤖 AI Added Allow: /blog/how-coinbase-used-code-connect-to-shrink-token-costs/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/start-anywhere/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +9 −0
  • 🤖 AI Added Allow: /blog/how-we-built-generative-plugins-and-shaders/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/sightlines-issue-no2/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/sightlines-issue-no2/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/sightlines-issue-no2/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/sightlines-issue-no2/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/sightlines-issue-no2/$ for CHATGPT-User
  • … 39 more change(s)
view diff → snapshot
Roblox roblox.com modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Europe PMC europepmc.org modified +0 −141
  • Removed all rules for user-agent ‘baiduspider’
  • Removed all rules for user-agent ‘bingbot’
  • 🤖 AI Removed rules for AI crawler ChatGPT-User (OpenAI)
  • Removed all rules for user-agent ‘Googlebot’
  • Removed all rules for user-agent ‘MSNbot’
  • 🤖 AI Removed rules for AI crawler OAI-SearchBot (OpenAI)
  • … 19 more change(s)
view diff → snapshot
Europe PMC europepmc.org modified +141 −0
  • Added rules for user-agent ‘baiduspider’
  • Added rules for user-agent ‘bingbot’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • Added rules for user-agent ‘Googlebot’
  • Added rules for user-agent ‘MSNbot’
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • … 18 more change(s)
view diff → snapshot
Europe PMC europepmc.org modified +0 −141
  • Removed all rules for user-agent ‘baiduspider’
  • Removed all rules for user-agent ‘bingbot’
  • 🤖 AI Removed rules for AI crawler ChatGPT-User (OpenAI)
  • Removed all rules for user-agent ‘Googlebot’
  • Removed all rules for user-agent ‘MSNbot’
  • 🤖 AI Removed rules for AI crawler OAI-SearchBot (OpenAI)
  • … 19 more change(s)
view diff → snapshot
Figma Community figma.com modified +3 −0
  • 🤖 AI Added Allow: /blog/measuring-the-impact-of-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/expressive-design-tools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/measuring-the-impact-of-ai/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /webinars/expressive-design-tools/$ for Claude-SearchBot
  • … 9 more change(s)
view diff → snapshot
Europe PMC europepmc.org modified +141 −0
  • Added rules for user-agent ‘baiduspider’
  • Added rules for user-agent ‘bingbot’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • Added rules for user-agent ‘Googlebot’
  • Added rules for user-agent ‘MSNbot’
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • … 18 more change(s)
view diff → snapshot
Insight Enterprises insight.com modified +9 −137
  • 🤖 AI Unblocked anthropic-ai (Anthropic) - rules removed
  • Removed all rules for user-agent ‘bingbot’
  • 🤖 AI Unblocked ChatGPT-User (OpenAI) - rules removed
  • 🤖 AI Unblocked Claude-Web (Anthropic) - rules removed
  • 🤖 AI Unblocked cohere-ai (Cohere) - rules removed
  • Removed all rules for user-agent ‘Googlebot’
  • … 17 more change(s)
view diff → snapshot
Insight Enterprises insight.com modified +145 −8
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • Added rules for user-agent ‘bingbot’
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Blocked cohere-ai (Cohere) entirely
  • Added rules for user-agent ‘Googlebot’
  • … 15 more change(s)
view diff → snapshot
Tripadvisor tripadvisor.com modified +0 −2
  • Removed Disallow: /AIAssistant for *
  • Removed Disallow: /ProxyToolsAccountSettingsTool for *
  • Removed Disallow: /AIAssistant for applebot
  • Removed Disallow: /ProxyToolsAccountSettingsTool for applebot
  • Removed Disallow: /AIAssistant for bingbot
  • Removed Disallow: /ProxyToolsAccountSettingsTool for bingbot
  • … 12 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/workflow-lab-moving-between-design-and-code-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/workflow-lab-moving-between-design-and-code-with-agents/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/workflow-lab-moving-between-design-and-code-with-agents/$ for Claude-User
  • 🤖 AI Added Allow: /blog/workflow-lab-moving-between-design-and-code-with-agents/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/workflow-lab-moving-between-design-and-code-with-agents/$ for PerplexityBot
view diff → snapshot
USA Today usatoday.com modified +12 −0
  • Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT Operator
  • Added Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT Operator
  • 🤖 AI Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT-User
  • 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT-User
  • 🤖 AI Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for Google-Extended
  • 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for Google-Extended
  • … 6 more change(s)
view diff → snapshot
Toyota toyota.com modified +1 −15
  • Removed Disallow: /espanol/vehicles/ for *
  • Removed Disallow: /vehicles/ for *
  • Removed Disallow: /espanol/vehicles/ for Applebot
  • Removed Disallow: /vehicles/ for Applebot
  • 🤖 AI Removed Disallow: /espanol/vehicles/ for ChatGPT-User
  • 🤖 AI Removed Disallow: /vehicles/ for ChatGPT-User
  • … 8 more change(s)
view diff → snapshot
IGN ign.com modified +1 −1
  • Added Allow: /sponsored/ for "008"
  • Removed Allow: /sponsored/ for *
  • Added Allow: /sponsored/ for AddSearchBot
  • 🤖 AI Added Allow: /sponsored/ for AI2Bot
  • Added Allow: /sponsored/ for Ai2Bot-DeepResearchEval
  • Added Allow: /sponsored/ for Ai2Bot-Dolma
  • … 45 more change(s)
view diff → snapshot