NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Showing initial archives
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/the-tldr-on-figma-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/legal/privacy/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/the-tldr-on-figma-weave/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /pt-br/legal/privacy/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/the-tldr-on-figma-weave/$ for Claude-User
  • 🤖 AI Added Allow: /pt-br/legal/privacy/$ for Claude-User
  • … 4 more change(s)
view diff → snapshot
Universal Music Group universalmusic.com modified +1 −1185
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked GPTBot (OpenAI) - rules removed
  • Removed sitemap https://www.universalmusic.com/sitemap_index.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Lam Research lamresearch.com removed +0 −9
  • robots.txt is no longer served (HTTP 404)
Figma Community figma.com modified +6 −1
  • 🤖 AI Added Allow: /de-de/reports/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/reports/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/reports/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/reports/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/reports/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/introducing-weave-creative-workflow-building/$ for CHATGPT-User
  • … 29 more change(s)
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/solution-briefs/pfpt-us-sb-semantic-business-policies.pdf for *
view diff → snapshot
Markel Group markel.com removed +0 −18
  • robots.txt is no longer served (HTTP 404)
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +29 −2
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /marketplace/az/* for *
  • Added Disallow: /marketplace/ca/* for *
  • Added Disallow: /marketplace/fl/* for *
  • Added Disallow: /marketplace/ga/* for *
  • Added Disallow: /marketplace/ia/* for *
  • … 23 more change(s)
view diff → snapshot
Petronas petronas.com removed +0 −61
  • robots.txt is no longer served (HTTP 404)
Pixiv pixiv.net modified +1 −0
  • 🤖 AI Added Allow: /info.php for anthropic-ai
  • 🤖 AI Added Allow: /info.php for Applebot-Extended
  • 🤖 AI Added Allow: /info.php for CCBot
  • 🤖 AI Added Allow: /info.php for ClaudeBot
  • 🤖 AI Added Allow: /info.php for cohere-ai
  • 🤖 AI Added Allow: /info.php for Diffbot
  • … 6 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
The Asahi Shimbun asahi.com modified +22 −0
  • Added Allow: /corporate/ for *
  • 🤖 AI Added Allow: /corporate/ for AmazonBot
  • 🤖 AI Added Allow: /corporate/ for anthropic-ai
  • 🤖 AI Added Allow: /corporate/ for Applebot-Extended
  • 🤖 AI Added Allow: /corporate/ for ChatGPT-User
  • 🤖 AI Added Allow: /corporate/ for Claude-SearchBot
  • … 16 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/making-something-human-4-things-we-took-away-from-config-india/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/making-something-human-4-things-we-took-away-from-config-india/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/making-something-human-4-things-we-took-away-from-config-india/$ for Claude-User
  • 🤖 AI Added Allow: /blog/making-something-human-4-things-we-took-away-from-config-india/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/making-something-human-4-things-we-took-away-from-config-india/$ for PerplexityBot
view diff → snapshot