NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Hiding initial archives
Shell shell.com modified +2 −0
  • Added Disallow: /motorist/oils-lubricants/rimula-truck-heavy-duty-engine-oil/shell-starship/immersion-cooling-battery-technology.html for *
  • Added Disallow: /motorist/oils-lubricants/rimula-truck-heavy-duty-engine-oil/shell-starship/immersion-cooling-battery-technology/ for *
view diff → snapshot
Semrush semrush.com modified +1 −1
  • Added Allow: /analytics/keywordoverview/* for *
  • Removed Allow: /analytics/keywordoverview/*?q=* for *
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Thyssenkrupp thyssenkrupp.com modified +1 −9
  • Added Disallow: (empty) for *
  • Removed Disallow: *UCPthyssenkruppAG* for *
  • Removed Disallow: /_api for *
  • Removed Disallow: /_media/ for *
  • Removed Disallow: /assets.files/ for *
  • Removed Disallow: /de/home-2604 for *
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
University College London ucl.ac.uk modified +31 −296
  • Added sitemap https://www.ucl.ac.uk/about/sitemap.xml
  • Added sitemap https://www.ucl.ac.uk/alumni/sitemap.xml
  • Added sitemap https://www.ucl.ac.uk/arts-humanities/sitemap.xml
  • Added sitemap https://www.ucl.ac.uk/bartlett/sitemap.xml
  • Added sitemap https://www.ucl.ac.uk/brain-sciences/sitemap.xml
  • Added sitemap https://www.ucl.ac.uk/brand-and-experience/sitemap.xml
  • … 45 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Reliance Steel & Aluminum rsac.com modified +23 −7
  • Blocked user-agent ‘AhrefsBot’ entirely
  • Blocked user-agent ‘BLEXBot’ entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked Claude-SearchBot (Anthropic) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • … 15 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
DuckDuckGo duckduckgo.com modified +5 −4
  • Added Allow: /?* for *
  • Removed Disallow: /2012-privacy-policy for *
  • Added sitemap https://duckduckgo.com/duckduckgo-help-pages/sitemap.xml
view diff → snapshot
Microsoft microsoft.com modified +19 −1
  • Added Allow: /?display=wide for *
  • Added Allow: /wp-admin/admin-ajax.php for *
  • Added Allow: /wp-content/uploads/ for *
  • Added Disallow: /*/embed/ for *
  • Added Disallow: /*?*p=* for *
  • Added Disallow: /*?p= for *
  • … 12 more change(s)
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
The Trade Desk thetradedesk.com modified +1 −1
  • Added sitemap https://careers.thetradedesk.com/sitemap.xml
  • Removed sitemap https://www.thetradedesk.com/sitemap.xml
view diff → snapshot
Cummins cummins.com modified +1 −1
  • Added Disallow: /*/mp-resource/* for *
  • Removed Disallow: /*/mp-resource/en-na/* for *
view diff → snapshot