NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2609snapshots
1437changes detected
226this week
All changes 🤖 AI-crawler changes Hiding initial archives
BBC bbc.com modified +3 −2
  • Added sitemap https://www.bbc.com/sitemaps/world-service-homepages.xml
view diff → snapshot
Barracuda barracuda.com removed +0 −232
  • robots.txt is no longer served (HTTP 404)
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Figma Community figma.com modified +43 −0
  • 🤖 AI Added Allow: /blog/sightlines-issue-no1/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/figma-make-general-availability/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/future-states/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/gpt-5-6-is-now-available-in-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/figma-make-general-availability/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/future-states/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Europe PMC europepmc.org modified +98 −7
  • Added rules for user-agent ‘baiduspider’
  • Added rules for user-agent ‘bingbot’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
  • … 21 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Safran safran-group.com modified +6 −0
  • Added Disallow: /cn/contact/vcf/ for *
  • Added Disallow: /contact/vcf/ for *
  • Added Disallow: /es/contact/vcf/ for *
  • Added Disallow: /fr/contact/vcf/ for *
  • Added Disallow: /hi/contact/vcf/ for *
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/workflow-lab-deploying-designs-directly-with-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/design-explore/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/workflow-lab-deploying-designs-directly-with-figma-make/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /es-la/design-explore/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/workflow-lab-deploying-designs-directly-with-figma-make/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /es-la/design-explore/$ for OAI-SearchBot
  • … 2 more change(s)
view diff → snapshot
Ford Motor ford.com modified +46 −0
  • Blocked user-agent ‘amazon-kendra’ entirely
  • Blocked user-agent ‘amazon-QBusiness’ entirely
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Blocked user-agent ‘bedrockbot’ entirely
  • … 24 more change(s)
view diff → snapshot
Europe PMC europepmc.org modified +7 −98
  • Removed all rules for user-agent ‘baiduspider’
  • Removed all rules for user-agent ‘bingbot’
  • 🤖 AI Removed rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Removed rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Removed rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Removed rules for AI crawler PerplexityBot (Perplexity)
  • … 21 more change(s)
view diff → snapshot