NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Hiding initial archives
IMDb imdb.com modified +11 −1
  • Removed Disallow: /*/contribute/* for AmazonAdBot
  • Removed Disallow: /*/register for AmazonAdBot
  • Removed Disallow: /*/registration/ for AmazonAdBot
  • Removed Disallow: /*/title/*/review/* for AmazonAdBot
  • Removed Disallow: /*/title/*/reviews for AmazonAdBot
  • Removed Disallow: /*/user/ur*/reviews for AmazonAdBot
  • … 6 more change(s)
view diff → snapshot
Insight Enterprises insight.com modified +9 −137
  • 🤖 AI Unblocked anthropic-ai (Anthropic) - rules removed
  • Removed all rules for user-agent ‘bingbot’
  • 🤖 AI Unblocked ChatGPT-User (OpenAI) - rules removed
  • 🤖 AI Unblocked Claude-Web (Anthropic) - rules removed
  • 🤖 AI Unblocked cohere-ai (Cohere) - rules removed
  • Removed all rules for user-agent ‘Googlebot’
  • … 17 more change(s)
view diff → snapshot
Electronic Arts ea.com modified +10 −0
  • Added Disallow: /*/games/the-sims/the-sims-4/home/ for *
  • Added Disallow: /*/games/the-sims/the-sims-4/store-pages/loginredirect for *
  • Added Disallow: /*/games/the-sims/the-sims-4/store-pages/logoutredirect for *
  • Added Disallow: /*/games/the-sims/the-sims-4/store-pages/paymentredirect for *
  • Added Disallow: /*/games/the-sims/the-sims-4/store-public-api/ for *
  • Added Disallow: /games/the-sims/the-sims-4/home/ for *
  • … 4 more change(s)
view diff → snapshot
Barracuda barracuda.com modified +133 −143
  • Added Disallow: /Sii8yP_qTEmmJiKsPYF7hZ21UiFM2l1ShyELGEXtZ9o=.html for *
  • Removed Disallow: /GqS-MCUe_qfcMaNsUdAZhx3lvVU0SOEjrSLgx8DrQ9M=.html for *
  • Added Disallow: /Sii8yP_qTEmmJiKsPYF7hZ21UiFM2l1ShyELGEXtZ9o=.html for DotBot
  • Removed Disallow: /GqS-MCUe_qfcMaNsUdAZhx3lvVU0SOEjrSLgx8DrQ9M=.html for DotBot
  • Added Disallow: /Sii8yP_qTEmmJiKsPYF7hZ21UiFM2l1ShyELGEXtZ9o=.html for sistrix
  • Removed Disallow: /GqS-MCUe_qfcMaNsUdAZhx3lvVU0SOEjrSLgx8DrQ9M=.html for sistrix
  • … 2 more change(s)
view diff → snapshot
Insight Enterprises insight.com modified +145 −8
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • Added rules for user-agent ‘bingbot’
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Blocked cohere-ai (Cohere) entirely
  • Added rules for user-agent ‘Googlebot’
  • … 15 more change(s)
view diff → snapshot
eBay ebay.com modified +3 −15
  • Blocked user-agent ‘OAI-AdsBot’ entirely
  • Added Allow: /itm/*?*mkcid=32* for OAI-AdsBot
  • Removed Disallow: /act/ for OAI-AdsBot
  • Removed Disallow: /adchoice for OAI-AdsBot
  • Removed Disallow: /fdbk/ for OAI-AdsBot
  • Removed Disallow: /myb for OAI-AdsBot
  • … 10 more change(s)
view diff → snapshot
Vistra vistra.com modified +4 −0
  • Added Disallow: /*/media/oembed for *
  • Added Disallow: /media/oembed for *
  • Added Disallow: /*/media/oembed for Googlebot-News
  • Added Disallow: /media/oembed for Googlebot-News
view diff → snapshot
Tripadvisor tripadvisor.com modified +0 −2
  • Removed Disallow: /AIAssistant for *
  • Removed Disallow: /ProxyToolsAccountSettingsTool for *
  • Removed Disallow: /AIAssistant for applebot
  • Removed Disallow: /ProxyToolsAccountSettingsTool for applebot
  • Removed Disallow: /AIAssistant for bingbot
  • Removed Disallow: /ProxyToolsAccountSettingsTool for bingbot
  • … 12 more change(s)
view diff → snapshot
F5 f5.com removed +0 −10
  • robots.txt is no longer served (HTTP 404)
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/workflow-lab-moving-between-design-and-code-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/workflow-lab-moving-between-design-and-code-with-agents/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/workflow-lab-moving-between-design-and-code-with-agents/$ for Claude-User
  • 🤖 AI Added Allow: /blog/workflow-lab-moving-between-design-and-code-with-agents/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/workflow-lab-moving-between-design-and-code-with-agents/$ for PerplexityBot
view diff → snapshot
Nextdoor nextdoor.com modified +24 −0
  • Added rules for user-agent ‘bingbot’
  • Added rules for user-agent ‘DuckDuckBot’
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
USA Today usatoday.com modified +12 −0
  • Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT Operator
  • Added Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT Operator
  • 🤖 AI Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for ChatGPT-User
  • 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for ChatGPT-User
  • 🤖 AI Added Allow: /story/sponsor-story/address-usa/2026/08/21/should-sellers-accept-the-first-offer-what-to-consider-beyond-price/91390104007/ for Google-Extended
  • 🤖 AI Added Allow: /story/sponsor-story/real-estate-ausa/2026/08/21/buy-now-or-wait-until-fall-tips-for-homebuyers-in-2026/91389661007/ for Google-Extended
  • … 6 more change(s)
view diff → snapshot