NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Showing initial archives
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /de-de/blog/increased-ipo-price-range/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/london-office-expansion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/london-office-expansion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/london-office-expansion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/london-office-expansion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/london-office-expansion/$ for CHATGPT-User
  • … 29 more change(s)
view diff → snapshot
General Motors gm.com modified +133 −53
  • Added rules for user-agent ‘AdobeEdgeOptimize/1.0’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • Added rules for user-agent ‘Googlebot-Mobile’
  • Added rules for user-agent ‘Googlebot-PageRenderer’
  • Added rules for user-agent ‘GoogleMessages’
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/workflow-lab-from-design-system-to-campaign-in-figma-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/workflow-lab-from-design-system-to-campaign-in-figma-weave/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/workflow-lab-from-design-system-to-campaign-in-figma-weave/$ for Claude-User
  • 🤖 AI Added Allow: /blog/workflow-lab-from-design-system-to-campaign-in-figma-weave/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/workflow-lab-from-design-system-to-campaign-in-figma-weave/$ for PerplexityBot
view diff → snapshot
Check Point checkpoint.com modified +31 −0
  • Added rules for user-agent ‘AndiBot’
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • Added rules for user-agent ‘ExaBot’
  • … 7 more change(s)
view diff → snapshot
Figma Community figma.com modified +8 −0
  • 🤖 AI Added Allow: /de-de/blog/the-power-of-product-based-learning/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/the-power-of-product-based-learning/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/the-power-of-product-based-learning/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/the-power-of-product-based-learning/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/the-power-of-product-based-learning/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/the-power-of-product-based-learning/$ for CHATGPT-User
  • … 34 more change(s)
view diff → snapshot
SoundCloud soundcloud.com modified +29 −60
  • Blocked user-agent ‘Brightbot’ entirely
  • 🤖 AI Blocked omgili (Webz.io) entirely
  • Blocked user-agent ‘omgilibot’ entirely
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +5 −0
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Added Allow: /customers/how-adyen-is-rebuilding-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/business-design-education/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/how-adyen-is-rebuilding-design/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /ko-kr/blog/business-design-education/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /customers/how-adyen-is-rebuilding-design/$ for Claude-User
  • … 5 more change(s)
view diff → snapshot
EDF edf.fr modified +86 −40
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • Added rules for user-agent ‘Applebot’
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • … 18 more change(s)
view diff → snapshot
SoundCloud soundcloud.com modified +60 −29
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Blocked cohere-ai (Cohere) entirely
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +21 −1
  • 🤖 AI Added Allow: /resource-library/agentic-ai-for-content-creation/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/ai-cinematography/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/ai-short-film-prompts/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/color-grading/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/compositing/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/lip-sync-animation/$ for CHATGPT-User
  • … 34 more change(s)
view diff → snapshot
Kimberly-Clark kimberly-clark.com modified +109 −2
  • Added rules for user-agent ‘AndiBot’
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • Added rules for user-agent ‘Applebot’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Added rules for user-agent ‘Bingbot’
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • … 45 more change(s)
view diff → snapshot
Der Spiegel spiegel.de modified +36 −6
  • 🤖 AI Added Allow: /ausland/ for Claude-SearchBot
  • 🤖 AI Added Allow: /familie/ for Claude-SearchBot
  • 🤖 AI Added Allow: /gesundheit/ for Claude-SearchBot
  • 🤖 AI Added Allow: /partnerschaft/ for Claude-SearchBot
  • 🤖 AI Added Allow: /psychologie/ for Claude-SearchBot
  • 🤖 AI Added Allow: /reise/ for Claude-SearchBot
  • … 18 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/london-office-expansion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/london-office-expansion/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/london-office-expansion/$ for Claude-User
  • 🤖 AI Added Allow: /blog/london-office-expansion/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/london-office-expansion/$ for PerplexityBot
view diff → snapshot
Kimberly-Clark kimberly-clark.com modified +2 −109
  • Removed all rules for user-agent ‘AndiBot’
  • 🤖 AI Unblocked anthropic-ai (Anthropic) - rules removed
  • Removed all rules for user-agent ‘Applebot’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • Removed all rules for user-agent ‘Bingbot’
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
USA Today usatoday.com modified +27 −6
  • Added rules for user-agent ‘NewsBank.com’
  • Added Allow: /story/branded-story/ for ChatGPT Operator
  • Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT Operator
  • Added Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT Operator
  • 🤖 AI Added Allow: /story/branded-story/ for ChatGPT-User
  • 🤖 AI Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT-User
  • … 13 more change(s)
view diff → snapshot
Expedia Group expedia.com modified +6 −0
  • Added Disallow: /cl/* for *
  • Added Disallow: /cl/* for AdIdxBot
  • Added Disallow: /cl/* for AdsBot-Google
  • Added Disallow: /cl/* for AdsBot-Google-Mobile
  • 🤖 AI Added Disallow: /cl/* for ChatGPT-User
  • 🤖 AI Added Disallow: /cl/* for Claude-SearchBot
  • … 7 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Yandex yandex.ru modified +4 −4
  • Added Disallow: /alice*? for *
  • Removed Disallow: /alice/*? for *
  • Added Disallow: /alice*? for Bingbot
  • Removed Disallow: /alice/*? for Bingbot
  • Added Disallow: /alice*? for Googlebot
  • Removed Disallow: /alice/*? for Googlebot
  • … 2 more change(s)
view diff → snapshot
Bloomberg bloomberg.com modified +0 −38
  • 🤖 AI Removed Allow: /$ for ClaudeBot
  • 🤖 AI Removed Allow: /africa$ for ClaudeBot
  • 🤖 AI Removed Allow: /africa/$ for ClaudeBot
  • 🤖 AI Removed Allow: /asia$ for ClaudeBot
  • 🤖 AI Removed Allow: /asia/$ for ClaudeBot
  • 🤖 AI Removed Allow: /canada$ for ClaudeBot
  • … 32 more change(s)
view diff → snapshot
Sempra sempra.com modified +78 −58
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 45 more change(s)
view diff → snapshot
Airbnb airbnb.com modified +24 −134
  • 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Blocked user-agent ‘MistralAI-Training’ entirely
  • Blocked user-agent ‘Webzio-Extended’ entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Removed Allow: /.well-known/amphtml/apikey.pub for ClaudeBot
  • … 45 more change(s)
view diff → snapshot
C.H. Robinson chrobinson.com modified +6 −0
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • Added Disallow: Amazonbot for *
  • Added Disallow: Amzn-SearchBot for *
  • Added Disallow: Bytespider for *
view diff → snapshot
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /de-de/blog/source-material-aitana-pico/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-aitana-pico/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-aitana-pico/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/source-material-aitana-pico/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/source-material-aitana-pico/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/source-material-aitana-pico/$ for CHATGPT-User
  • … 29 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/what-it-takes-to-build-great-products-now/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/what-it-takes-to-build-great-products-now/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/what-it-takes-to-build-great-products-now/$ for Claude-User
  • 🤖 AI Added Allow: /blog/what-it-takes-to-build-great-products-now/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/what-it-takes-to-build-great-products-now/$ for PerplexityBot
view diff → snapshot
Figma Community figma.com modified +5 −0
  • 🤖 AI Added Allow: /blog/the-power-of-product-based-learning/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/reports/ai-native-organization/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/reports/ai-native-organization/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/reports/ai-native-organization/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/reports/ai-native-organization/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/the-power-of-product-based-learning/$ for Claude-SearchBot
  • … 19 more change(s)
view diff → snapshot
Reuters reuters.com modified +1 −0
  • Added Disallow: /*/site-search/ for AASA-Bot
  • Added Disallow: /*/site-search/ for ADmantX
  • Added Disallow: /*/site-search/ for AdmiralBot
  • Added Disallow: /*/site-search/ for AdsBot-Google
  • Added Disallow: /*/site-search/ for AdsBot-Google-Mobile
  • Added Disallow: /*/site-search/ for AmazonAdBot
  • … 45 more change(s)
view diff → snapshot
Daily Mail dailymail.co.uk modified +1 −2
  • Removed Allow: /buyline/ for AhrefsBot
  • 🤖 AI Removed Allow: /buyline/ for AI2Bot
  • Removed Allow: /buyline/ for Ai2Bot-Dolma
  • Removed Allow: /buyline/ for amazon-kendra-
  • 🤖 AI Removed Allow: /buyline/ for Amazonbot
  • 🤖 AI Removed Allow: /buyline/ for anthropic-ai
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +10 −0
  • 🤖 AI Added Allow: /es-es/reports/ai-native-organization/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/double-click-does-efficiency-kill-love/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-zach-lieberman/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/reports/ai-native-organization/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/reports/jp-developers-field-guide-ai-era/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/reports/kr-developers-field-guide-ai-era/$ for CHATGPT-User
  • … 44 more change(s)
view diff → snapshot
Bloomberg bloomberg.com modified +168 −42
  • 🤖 AI Added Allow: /$ for ChatGPT-User
  • 🤖 AI Added Allow: /africa$ for ChatGPT-User
  • 🤖 AI Added Allow: /africa/$ for ChatGPT-User
  • 🤖 AI Added Allow: /asia$ for ChatGPT-User
  • 🤖 AI Added Allow: /asia/$ for ChatGPT-User
  • 🤖 AI Added Allow: /canada$ for ChatGPT-User
  • … 45 more change(s)
view diff → snapshot