NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
Daily Mail dailymail.co.uk modified +5 −1
  • Blocked user-agent ‘cohere-training-data-crawler’ entirely
  • Blocked user-agent ‘GPTBot-Search’ entirely
  • 🤖 AI Blocked Meta-WebIndexer (Meta) entirely
  • Blocked user-agent ‘xAI-SearchBot’ entirely
view diff → snapshot
Frontier Communications frontier.com modified +1 −190
  • 🤖 AI Removed rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Removed rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Removed rules for AI crawler Google-Extended (Google)
  • 🤖 AI Removed rules for AI crawler GPTBot (OpenAI)
  • Removed all rules for user-agent ‘OAI-AdsBot’
  • 🤖 AI Removed rules for AI crawler OAI-SearchBot (OpenAI)
  • … 1 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +28 −3
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /marketplace/az/* for *
  • Added Disallow: /marketplace/ca/* for *
  • Added Disallow: /marketplace/fl/* for *
  • Added Disallow: /marketplace/ga/* for *
  • Added Disallow: /marketplace/ia/* for *
  • … 24 more change(s)
view diff → snapshot
Zillow zillow.com modified +265 −0
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
view diff → snapshot
Nikkei nikkei.com modified +24 −21
  • 🤖 AI Added Allow: /promotion/$ for ChatGPT-User
  • 🤖 AI Added Allow: /promotion/about/$ for ChatGPT-User
  • 🤖 AI Added Allow: /promotion/campaign/student/startguide/$ for ChatGPT-User
  • 🤖 AI Added Allow: /promotion/family/$ for ChatGPT-User
  • 🤖 AI Added Allow: /promotion/houjin/$ for ChatGPT-User
  • 🤖 AI Added Allow: /promotion/houjin/casestudies/ for ChatGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/source-material-brent-david-freaney/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-brent-david-freaney/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-brent-david-freaney/$ for Claude-User
  • 🤖 AI Added Allow: /blog/source-material-brent-david-freaney/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-brent-david-freaney/$ for PerplexityBot
view diff → snapshot
Owens Corning owenscorning.com modified +32 −0
  • Added rules for user-agent ‘*’
  • Added sitemap https://www.owenscorning.com/sitemap-1.xml
  • Added sitemap https://www.owenscorning.com/sitemap.xml
view diff → snapshot
Zillow zillow.com modified +5 −0
  • Added Disallow: /auth/ for *
  • Added Disallow: /rental-manager-api/ for *
  • Added Disallow: /rental/capps/api/ for *
  • Added Disallow: /rental/satellite/api/ for *
  • Added Disallow: /rental/z-calendar/api/ for *
view diff → snapshot
Nubank nubank.com.br modified +30 −36
  • Removed Disallow: /nu/cartao/nu-limite-garantido for *
  • Removed Disallow: /nu/cartao/nu-limite-garantido for DatoCmsSearchBot
view diff → snapshot
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /customers/how-theodo-reinvents-the-product-workflow-with-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/workflow-lab-staying-in-the-flow-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/workflow-lab-staying-in-the-flow-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/workflow-lab-staying-in-the-flow-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/workflow-lab-staying-in-the-flow-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/customers/how-theodo-reinvents-the-product-workflow-with-figma/$ for CHATGPT-User
  • … 29 more change(s)
view diff → snapshot
The Atlantic theatlantic.com modified +90 −0
  • Added Disallow: /4624/TheAtlanticOnline/* for AmazonAdBot
  • Added Disallow: /ab/* for AmazonAdBot
  • Added Disallow: /search/?*q=* for AmazonAdBot
  • Added Disallow: /video/embed/ for AmazonAdBot
  • Added Disallow: /video/iframe/* for AmazonAdBot
  • Added Disallow: /zephr/* for AmazonAdBot
  • … 45 more change(s)
view diff → snapshot
Estee Lauder elcompanies.com modified +10 −0
  • Added Disallow: /en-gb/capital-markets-experience-2026 for *
  • Added Disallow: /en-gb/capital-markets-experience-2026/* for *
  • Added Disallow: /en/capital-markets-experience-2026 for *
  • Added Disallow: /en/capital-markets-experience-2026/* for *
  • Added Disallow: /es/capital-markets-experience-2026 for *
  • Added Disallow: /es/capital-markets-experience-2026/* for *
  • … 4 more change(s)
view diff → snapshot
Owens Corning owenscorning.com modified +0 −32
  • Removed all rules for user-agent ‘*’
  • Removed sitemap https://www.owenscorning.com/sitemap-1.xml
  • Removed sitemap https://www.owenscorning.com/sitemap.xml
view diff → snapshot