NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
All changes 🤖 AI-crawler changes Hiding initial archives
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Bloomberg bloomberg.com modified +0 −38
  • 🤖 AI Removed Allow: /$ for ClaudeBot
  • 🤖 AI Removed Allow: /africa$ for ClaudeBot
  • 🤖 AI Removed Allow: /africa/$ for ClaudeBot
  • 🤖 AI Removed Allow: /asia$ for ClaudeBot
  • 🤖 AI Removed Allow: /asia/$ for ClaudeBot
  • 🤖 AI Removed Allow: /canada$ for ClaudeBot
  • … 32 more change(s)
view diff → snapshot
Semrush semrush.com modified +1 −3
  • Added Disallow: /_/semrush_regular_1_blank/ for *
  • Removed Disallow: /_/empty_blank/ for *
  • Removed Disallow: /_/site_info_blank/ for *
  • Removed Disallow: /_/toolkit_blank/ for *
view diff → snapshot
Goodyear Tire & Rubber goodyear.com modified +40 −0
  • Blocked user-agent ‘ahrefsbot’ entirely
  • Blocked user-agent ‘cazoodlebot’ entirely
  • Blocked user-agent ‘dotbot/1.0’ entirely
  • Blocked user-agent ‘gigabot’ entirely
  • Blocked user-agent ‘mj12bot’ entirely
  • Blocked user-agent ‘sogou inst spider’ entirely
  • … 20 more change(s)
view diff → snapshot
Sweden (Regeringen) government.se modified +2 −2
  • Added Disallow: /sokresultat/?query= for *
  • Removed Disallow: /search/?query= for *
  • Added sitemap https://www.regeringen.se/sitemap.xml
  • Removed sitemap https://www.government.se/en/sitemap.xml
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Telstra telstra.com.au modified +1 −0
  • Added Disallow: /content/dam/tcom/personal/consumer-advice/pdf/branded-calling-product-terms.pdf for *
view diff → snapshot
Goodyear Tire & Rubber goodyear.com modified +0 −40
  • Removed all rules for user-agent ‘ahrefsbot’
  • Removed all rules for user-agent ‘cazoodlebot’
  • Removed all rules for user-agent ‘dotbot/1.0’
  • Removed all rules for user-agent ‘gigabot’
  • Removed all rules for user-agent ‘mj12bot’
  • Removed all rules for user-agent ‘sogou inst spider’
  • … 20 more change(s)
view diff → snapshot
Sweden (Regeringen) government.se modified +2 −2
  • Added Disallow: /search/?query= for *
  • Removed Disallow: /sokresultat/?query= for *
  • Added sitemap https://www.government.se/en/sitemap.xml
  • Removed sitemap https://www.regeringen.se/sitemap.xml
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Shell shell.com modified +6 −14
  • Added Disallow: /shellenergy/othersolutions.html for *
  • Added Disallow: /shellenergy/othersolutions/ for *
  • Added Disallow: /shellenergy/shell-energy-europe/natural-gas-portfolio.html for *
  • Added Disallow: /shellenergy/shell-energy-europe/natural-gas-portfolio/ for *
  • Added Disallow: /what-we-do/renewable-power.html for *
  • Added Disallow: /what-we-do/renewable-power/ for *
  • … 14 more change(s)
view diff → snapshot
Datadog datadoghq.com modified +0 −4
  • Removed sitemap https://www.datadoghq.com/community/sitemap.xml
  • Removed sitemap https://www.datadoghq.com/newsletter/sitemap.xml
  • Removed sitemap https://www.datadoghq.com/page/sitemap.xml
  • Removed sitemap https://www.datadoghq.com/products/sitemap.xml
view diff → snapshot
Datadog datadoghq.com modified +4 −0
  • Added sitemap https://www.datadoghq.com/community/sitemap.xml
  • Added sitemap https://www.datadoghq.com/newsletter/sitemap.xml
  • Added sitemap https://www.datadoghq.com/page/sitemap.xml
  • Added sitemap https://www.datadoghq.com/products/sitemap.xml
view diff → snapshot