NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Showing initial archives
Shell shell.com modified +2 −0
  • Added Disallow: /terms-of-use/shell-motorsports-ferrari-giveaway-activity-terms-and-conditions.html for *
  • Added Disallow: /terms-of-use/shell-motorsports-ferrari-giveaway-activity-terms-and-conditions/ for *
view diff → snapshot
Jabil jabil.com modified +5 −1
  • Added Allow: /.resources/jabil-main/themes/jabil-one-26/src/img/favicon/ for *
view diff → snapshot
Canva canva.com modified +2 −0
  • 🤖 AI Added Allow: /*/design-school/ for anthropic-ai
  • 🤖 AI Added Allow: /design-school/ for anthropic-ai
  • 🤖 AI Added Allow: /*/design-school/ for Applebot-Extended
  • 🤖 AI Added Allow: /design-school/ for Applebot-Extended
  • 🤖 AI Added Allow: /*/design-school/ for CCBot
  • 🤖 AI Added Allow: /design-school/ for CCBot
  • … 6 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Best Buy bestbuy.com modified +0 −1
  • 🤖 AI Removed Disallow: */site/searchpage.jsp?st=* for AmazonBot
  • Removed Disallow: */site/searchpage.jsp?st=* for Applebot
  • 🤖 AI Removed Disallow: */site/searchpage.jsp?st=* for ChatGPT-User
  • 🤖 AI Removed Disallow: */site/searchpage.jsp?st=* for Claude-User
  • 🤖 AI Removed Disallow: */site/searchpage.jsp?st=* for Claudebot
  • Removed Disallow: */site/searchpage.jsp?st=* for Gemeni-AI
  • … 6 more change(s)
view diff → snapshot
Algolia algolia.com modified +3 −0
  • Added sitemap https://www.algolia.com/doc/sitemap.xml
  • Added sitemap https://www.algolia.com/sitemap.xml
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /es-la/blog/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/start-anywhere/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /es-la/blog/start-anywhere/$ for Claude-User
  • 🤖 AI Added Allow: /es-la/blog/start-anywhere/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /es-la/blog/start-anywhere/$ for PerplexityBot
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Deutsche Telekom telekom.com modified +2 −8
  • Removed Disallow: /*.ics$ for *
  • Removed Disallow: /ajax/ for *
  • Removed Disallow: /de/glossar-details/ for *
  • Removed Disallow: /dynamic/fragment/ for *
  • Removed Disallow: /en/glossary-details/ for *
  • Removed Disallow: /ui-com-16-de/ for *
  • … 4 more change(s)
view diff → snapshot