NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Showing initial archives
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/6-ways-teams-are-experimenting-with-code-layers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/6-ways-teams-are-experimenting-with-code-layers/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/6-ways-teams-are-experimenting-with-code-layers/$ for Claude-User
  • 🤖 AI Added Allow: /blog/6-ways-teams-are-experimenting-with-code-layers/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/6-ways-teams-are-experimenting-with-code-layers/$ for PerplexityBot
view diff → snapshot
Figma Community figma.com modified +46 −0
  • 🤖 AI Added Allow: /de-de/blog/4-team-rituals-that-drive-ai-adoption/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/design-together-with-code-layers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/measuring-the-impact-of-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/downloads/chrome-extension/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/4-team-rituals-that-drive-ai-adoption/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/design-together-with-code-layers/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
SentinelOne sentinelone.com modified +16 −19
  • Added Allow: / for *
  • Added Disallow: /api/ for *
  • Added sitemap https://www.sentinelone.com/de/sitemap-index.xml
  • Added sitemap https://www.sentinelone.com/es/sitemap-index.xml
  • Added sitemap https://www.sentinelone.com/fr/sitemap-index.xml
  • Added sitemap https://www.sentinelone.com/it/sitemap-index.xml
  • … 13 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Xylem xylem.com modified +1 −1
  • Removed Disallow: /es-la/ for *
  • Added sitemap https://www.xylem.com/es-la/sitemap.xml
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +2 −29
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /marketplace/az/* for *
  • Removed Disallow: /marketplace/ca/* for *
  • Removed Disallow: /marketplace/fl/* for *
  • Removed Disallow: /marketplace/ga/* for *
  • … 23 more change(s)
view diff → snapshot
Nubank nubank.com.br modified +12 −6
  • Added Disallow: /contatos/ouvidoria-formulario-fechado for *
  • Added Disallow: /contatos/ouvidoria-formulario-fechado for DatoCmsSearchBot
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +29 −2
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /marketplace/az/* for *
  • Added Disallow: /marketplace/ca/* for *
  • Added Disallow: /marketplace/fl/* for *
  • Added Disallow: /marketplace/ga/* for *
  • Added Disallow: /marketplace/ia/* for *
  • … 23 more change(s)
view diff → snapshot
CME Group cmegroup.com modified +2 −1
  • Removed Disallow: /tools-information/webhelp/cme-direct/ for *
  • Added sitemap https://www.cmegroup.com/eds/sitemap.xml
  • Added sitemap https://www.cmegroup.com/sitemap.xml
view diff → snapshot