NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
227this week
All changes 🤖 AI-crawler changes Hiding initial archives
Bar-Ilan University biu.ac.il removed +0 −207
  • robots.txt is no longer served (HTTP 404)
GitHub github.com modified +140 −9
  • Added rules for user-agent ‘AdsBot-Google’
  • Added rules for user-agent ‘AdsBot-Google-Mobile’
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • … 2 more change(s)
view diff → snapshot
Zscaler zscaler.com modified +1 −1
  • 🤖 AI Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for Applebot-Extended
  • 🤖 AI Removed Disallow: /indexNowKey.txt for Applebot-Extended
  • 🤖 AI Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for ChatGPT-User
  • 🤖 AI Removed Disallow: /indexNowKey.txt for ChatGPT-User
  • 🤖 AI Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for Claude-SearchBot
  • 🤖 AI Removed Disallow: /indexNowKey.txt for Claude-SearchBot
  • … 14 more change(s)
view diff → snapshot
Nordstrom nordstrom.com modified +7 −1
  • Added rules for user-agent ‘meta-externalagent/1.1’
  • Added rules for user-agent ‘meta-webindexer/1.1’
view diff → snapshot
Costa Rica (Presidencia) presidencia.go.cr modified +0 −72
  • Removed Allow: /core/*.css$ for *
  • Removed Allow: /core/*.css? for *
  • Removed Allow: /core/*.gif for *
  • Removed Allow: /core/*.jpeg for *
  • Removed Allow: /core/*.jpg for *
  • Removed Allow: /core/*.js$ for *
  • … 45 more change(s)
view diff → snapshot
Bar-Ilan University biu.ac.il removed +0 −207
  • robots.txt is no longer served (HTTP 404)
Figma Community figma.com modified +5 −2
  • 🤖 AI Added Allow: /blog/japan-local-data-hosting/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /id-id/resource-library/$ for CHATGPT-User
  • 🤖 AI Added Allow: /th-th/resource-library/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /blog/source-material-creative-technologist-marisa-chentakul-meshtimes/$ for CHATGPT-User
  • … 29 more change(s)
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
TotalEnergies totalenergies.com modified +1 −1
  • Added sitemap https://totalenergies.com/sitemap.xml
  • Removed sitemap https://prd-backoffice.totalenergies.com/sitemap.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot