NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
Coursera coursera.org modified +94 −190
  • Added Allow: /courses?*&page=1$ for *
  • Added Allow: /courses?*&page=1& for *
  • Added Allow: /courses?*&page=2$ for *
  • Added Allow: /courses?*&page=2& for *
  • Added Allow: /courses?*&page=3$ for *
  • Added Allow: /courses?*&page=3& for *
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +29 −2
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /marketplace/az/* for *
  • Added Disallow: /marketplace/ca/* for *
  • Added Disallow: /marketplace/fl/* for *
  • Added Disallow: /marketplace/ga/* for *
  • Added Disallow: /marketplace/ia/* for *
  • … 23 more change(s)
view diff → snapshot
StoneX Group stonex.com modified +7 −8
  • Added Disallow: */search/?query= for *
  • Added sitemap https://www.stonex.com/ja-jp/sitemap-b2b.xml
  • Removed sitemap https://www.stonex.com/en/sitemap.xml
  • Removed sitemap https://www.stonex.com/pt-br/sitemap.xml
  • Removed sitemap https://www.stonex.com/sitemap.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Shell shell.com modified +2 −0
  • Added Disallow: /business-customers/industrial-lubricants-and-specialty-fluids-for-business/products/shell-panolin/about.html for *
  • Added Disallow: /business-customers/industrial-lubricants-and-specialty-fluids-for-business/products/shell-panolin/about/ for *
view diff → snapshot
GitHub github.com modified +342 −340
  • Added Disallow: /*/*/branches$ for *
  • Added Disallow: /*/*/branches/* for *
  • Added Disallow: /*/*/branches?* for *
  • Added Disallow: /*/*/comments$ for *
  • Added Disallow: /*/*/comments/* for *
  • Added Disallow: /*/*/comments?* for *
  • … 45 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +2 −29
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /marketplace/az/* for *
  • Removed Disallow: /marketplace/ca/* for *
  • Removed Disallow: /marketplace/fl/* for *
  • Removed Disallow: /marketplace/ga/* for *
  • … 23 more change(s)
view diff → snapshot
Kraken kraken.com modified +0 −1
  • Removed sitemap https://www.kraken.com/sitemaps/prices-asset-new-new-new.xml
view diff → snapshot
AT&T att.com modified +47 −26
  • Added Allow: /.well-known/api-catalog for *
  • Added Allow: /ai-instructions/build-a-plan.html for *
  • Added Allow: /ai-instructions/build-a-plan.md for *
  • Removed Allow: /.well-known/api-catalog.txt for *
  • Added Allow: /.well-known/agent-card.json for Googlebot
  • Added Allow: /.well-known/agent-skills/index.json for Googlebot
  • … 11 more change(s)
view diff → snapshot