NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5416snapshots
4280changes detected
237this week
All changes 🤖 AI-crawler changes Showing initial archives
BBC bbc.com modified +7 −1
  • 🤖 AI Blocked Claude-SearchBot (Anthropic) entirely
  • 🤖 AI Blocked DuckAssistBot (DuckDuckGo) entirely
view diff → snapshot
Qualys qualys.com modified +106 −1
  • Blocked user-agent ‘ExaSearchBot’ entirely
  • 🤖 AI Added rules for AI crawler meta-externalagent (Meta)
  • Added rules for user-agent ‘SemrushBot’
  • Added rules for user-agent ‘Serpstat’
  • Added sitemap https://www.qualys.com/sitemap-blog.xml
  • Removed sitemap https://blog.qualys.com/sitemap.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Reichman University runi.ac.il modified +29 −0
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler Google-Extended (Google)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler Perplexity-User (Perplexity)
  • … 1 more change(s)
view diff → snapshot
Universal Music Group universalmusic.com modified +1185 −1
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked GPTBot (OpenAI) entirely
  • Added sitemap https://www.universalmusic.com/sitemap_index.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Burlington Stores burlington.com modified +0 −69
  • 🤖 AI Unblocked anthropic-ai (Anthropic) - rules removed
  • Removed all rules for user-agent ‘Applebot’
  • Removed all rules for user-agent ‘AwarioRssBot’
  • Removed all rules for user-agent ‘AwarioSmartBot’
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • … 17 more change(s)
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/the-tldr-on-figma-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/legal/privacy/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/the-tldr-on-figma-weave/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /pt-br/legal/privacy/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/the-tldr-on-figma-weave/$ for Claude-User
  • 🤖 AI Added Allow: /pt-br/legal/privacy/$ for Claude-User
  • … 4 more change(s)
view diff → snapshot
Universal Music Group universalmusic.com modified +1 −1185
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked GPTBot (OpenAI) - rules removed
  • Removed sitemap https://www.universalmusic.com/sitemap_index.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +6 −1
  • 🤖 AI Added Allow: /de-de/reports/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/reports/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/reports/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/reports/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/reports/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/introducing-weave-creative-workflow-building/$ for CHATGPT-User
  • … 29 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Pixiv pixiv.net modified +1 −0
  • 🤖 AI Added Allow: /info.php for anthropic-ai
  • 🤖 AI Added Allow: /info.php for Applebot-Extended
  • 🤖 AI Added Allow: /info.php for CCBot
  • 🤖 AI Added Allow: /info.php for ClaudeBot
  • 🤖 AI Added Allow: /info.php for cohere-ai
  • 🤖 AI Added Allow: /info.php for Diffbot
  • … 6 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
The Asahi Shimbun asahi.com modified +22 −0
  • Added Allow: /corporate/ for *
  • 🤖 AI Added Allow: /corporate/ for AmazonBot
  • 🤖 AI Added Allow: /corporate/ for anthropic-ai
  • 🤖 AI Added Allow: /corporate/ for Applebot-Extended
  • 🤖 AI Added Allow: /corporate/ for ChatGPT-User
  • 🤖 AI Added Allow: /corporate/ for Claude-SearchBot
  • … 16 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/making-something-human-4-things-we-took-away-from-config-india/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/making-something-human-4-things-we-took-away-from-config-india/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/making-something-human-4-things-we-took-away-from-config-india/$ for Claude-User
  • 🤖 AI Added Allow: /blog/making-something-human-4-things-we-took-away-from-config-india/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/making-something-human-4-things-we-took-away-from-config-india/$ for PerplexityBot
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/training-a-figma-fluent-editing-model/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/training-a-figma-fluent-editing-model/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/training-a-figma-fluent-editing-model/$ for Claude-User
  • 🤖 AI Added Allow: /blog/training-a-figma-fluent-editing-model/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/training-a-figma-fluent-editing-model/$ for PerplexityBot
view diff → snapshot
Newsweek newsweek.com modified +6 −2
  • 🤖 AI Blocked Perplexity-User (Perplexity) entirely
  • Blocked user-agent ‘Shap-User’ entirely
  • Blocked user-agent ‘ShapBot’ entirely
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
view diff → snapshot
Delta Air Lines delta.com modified +67 −12
  • Added rules for user-agent ‘ai2-crawler’
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • … 31 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /customers/how-govtech-singapore-teams-use-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/how-govtech-singapore-teams-use-figma/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /customers/how-govtech-singapore-teams-use-figma/$ for Claude-User
  • 🤖 AI Added Allow: /customers/how-govtech-singapore-teams-use-figma/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /customers/how-govtech-singapore-teams-use-figma/$ for PerplexityBot
view diff → snapshot
Daily Mail dailymail.co.uk modified +5 −1
  • Blocked user-agent ‘cohere-training-data-crawler’ entirely
  • Blocked user-agent ‘GPTBot-Search’ entirely
  • 🤖 AI Blocked Meta-WebIndexer (Meta) entirely
  • Blocked user-agent ‘xAI-SearchBot’ entirely
view diff → snapshot
Frontier Communications frontier.com modified +1 −190
  • 🤖 AI Removed rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Removed rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Removed rules for AI crawler Google-Extended (Google)
  • 🤖 AI Removed rules for AI crawler GPTBot (OpenAI)
  • Removed all rules for user-agent ‘OAI-AdsBot’
  • 🤖 AI Removed rules for AI crawler OAI-SearchBot (OpenAI)
  • … 1 more change(s)
view diff → snapshot
Zillow zillow.com modified +265 −0
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
view diff → snapshot
Nikkei nikkei.com modified +24 −21
  • 🤖 AI Added Allow: /promotion/$ for ChatGPT-User
  • 🤖 AI Added Allow: /promotion/about/$ for ChatGPT-User
  • 🤖 AI Added Allow: /promotion/campaign/student/startguide/$ for ChatGPT-User
  • 🤖 AI Added Allow: /promotion/family/$ for ChatGPT-User
  • 🤖 AI Added Allow: /promotion/houjin/$ for ChatGPT-User
  • 🤖 AI Added Allow: /promotion/houjin/casestudies/ for ChatGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/source-material-brent-david-freaney/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-brent-david-freaney/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-brent-david-freaney/$ for Claude-User
  • 🤖 AI Added Allow: /blog/source-material-brent-david-freaney/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-brent-david-freaney/$ for PerplexityBot
view diff → snapshot
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /customers/how-theodo-reinvents-the-product-workflow-with-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/workflow-lab-staying-in-the-flow-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/workflow-lab-staying-in-the-flow-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/workflow-lab-staying-in-the-flow-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/workflow-lab-staying-in-the-flow-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/customers/how-theodo-reinvents-the-product-workflow-with-figma/$ for CHATGPT-User
  • … 29 more change(s)
view diff → snapshot
The Atlantic theatlantic.com modified +90 −0
  • Added Disallow: /4624/TheAtlanticOnline/* for AmazonAdBot
  • Added Disallow: /ab/* for AmazonAdBot
  • Added Disallow: /search/?*q=* for AmazonAdBot
  • Added Disallow: /video/embed/ for AmazonAdBot
  • Added Disallow: /video/iframe/* for AmazonAdBot
  • Added Disallow: /zephr/* for AmazonAdBot
  • … 45 more change(s)
view diff → snapshot