NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Hiding initial archives
TotalEnergies totalenergies.com modified +1 −1
  • Added sitemap https://prd-backoffice.totalenergies.com/sitemap.xml
  • Removed sitemap https://totalenergies.com/sitemap.xml
view diff → snapshot
KeyCorp key.com modified +128 −5
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • … 13 more change(s)
view diff → snapshot
L3Harris Technologies l3harris.com modified +6 −0
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Barracuda barracuda.com created +28 −28
  • Added Disallow: /j12zLcHN-c87jy_k8NEZy6HCpMVXD6-a7DAqYlZyqtQ=.html for *
  • Removed Disallow: /AhL8dV0Pb0uaG5WQP3DgESS1GUB2wnrlikHc2eAdbUE=.html for *
  • Added Disallow: /j12zLcHN-c87jy_k8NEZy6HCpMVXD6-a7DAqYlZyqtQ=.html for DotBot
  • Removed Disallow: /AhL8dV0Pb0uaG5WQP3DgESS1GUB2wnrlikHc2eAdbUE=.html for DotBot
  • Added Disallow: /j12zLcHN-c87jy_k8NEZy6HCpMVXD6-a7DAqYlZyqtQ=.html for sistrix
  • Removed Disallow: /AhL8dV0Pb0uaG5WQP3DgESS1GUB2wnrlikHc2eAdbUE=.html for sistrix
  • … 2 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Nvidia nvidia.com modified +22 −18
  • Added Disallow: /*accessToken=* for *
  • Added Disallow: /*cid=* for *
  • Added Disallow: /*eid=* for *
  • Added Disallow: /*hstc=* for *
  • Added Disallow: /*jso=* for *
  • Added Disallow: /*link=* for *
  • … 33 more change(s)
view diff → snapshot
Waldo waldo.fyi modified +2 −2
  • Added sitemap https://www.waldo.fyi/sitemap.xml
  • Removed sitemap https://waldo.fyi/sitemap.xml
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/try-these-10-skills-and-show-off-your-own/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/try-these-10-skills-and-show-off-your-own/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/try-these-10-skills-and-show-off-your-own/$ for Claude-User
  • 🤖 AI Added Allow: /blog/try-these-10-skills-and-show-off-your-own/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/try-these-10-skills-and-show-off-your-own/$ for PerplexityBot
view diff → snapshot
Barracuda barracuda.com removed +0 −232
  • robots.txt is no longer served (HTTP 404)