NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
227this week
All changes 🤖 AI-crawler changes Showing initial archives
BP bp.com modified +0 −2
  • Removed Disallow: /en_us/united-states/home/referencing.html for *
  • Removed Disallow: /en_us/united-states/home/referencing.html for Algolia Crawler
view diff → snapshot
Commonwealth Bank commbank.com.au modified +0 −2
  • Removed Disallow: /personal/eform/ for *
  • 🤖 AI Removed Disallow: /personal/eform/ for Amazonbot
  • 🤖 AI Removed Disallow: /personal/eform/ for Applebot-Extended
  • 🤖 AI Removed Disallow: /personal/eform/ for Bytespider
  • 🤖 AI Removed Disallow: /personal/eform/ for CCBot
  • 🤖 AI Removed Disallow: /personal/eform/ for ChatGPT-User
  • … 12 more change(s)
view diff → snapshot
Yandex yandex.ru modified +4 −0
  • Added Disallow: /alice/chat/ for *
  • Added Disallow: /alice/chat/ for Bingbot
  • Added Disallow: /alice/chat/ for Googlebot
  • Added Disallow: /alice/chat/ for Yandex
view diff → snapshot
Bar-Ilan University biu.ac.il removed +0 −207
  • robots.txt is no longer served (HTTP 404)
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/source-material-creative-technologist-marisa-chentakul-meshtimes/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-creative-technologist-marisa-chentakul-meshtimes/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-creative-technologist-marisa-chentakul-meshtimes/$ for Claude-User
  • 🤖 AI Added Allow: /blog/source-material-creative-technologist-marisa-chentakul-meshtimes/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-creative-technologist-marisa-chentakul-meshtimes/$ for PerplexityBot
view diff → snapshot
Snyk snyk.io modified +5 −0
  • Added Disallow: /*&topic= for *
  • Added Disallow: /*?topic= for *
view diff → snapshot
Moz moz.com modified +51 −4
  • Added rules for user-agent ‘AhrefsBot’
  • Added rules for user-agent ‘DDM-DCipher’
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • Added rules for user-agent ‘rogerbot’
  • Unblocked user-agent ‘*’
  • Added Allow: /learn/ for *
  • … 31 more change(s)
view diff → snapshot
Wealthsimple wealthsimple.com modified +3 −0
  • Added Disallow: /admin/* for *
  • Added Disallow: /invite for *
  • Added Disallow: /invite/* for *
view diff → snapshot
Tripadvisor tripadvisor.com modified +0 −2
  • Removed Disallow: /OwnerEmailManage for *
  • Removed Disallow: /developerslegacy for *
  • Removed Disallow: /OwnerEmailManage for applebot
  • Removed Disallow: /developerslegacy for applebot
  • Removed Disallow: /OwnerEmailManage for Baiduspider
  • Removed Disallow: /developerslegacy for Baiduspider
  • … 14 more change(s)
view diff → snapshot
AbbVie abbvie.com modified +7 −10
  • Added Disallow: /content/dam/corporate/abbvie-com2/pdfs/joinus for *
  • Added Disallow: /content/dam/corporate/abbvie-com2/pdfs/joinus/2024-total-rewards-resources-guide.pdf for *
  • Added Disallow: /content/dam/corporate/abbvie-com2/pdfs/joinus/2025-total-rewards-brochure-aug2025.pdf for *
  • Added Disallow: /content/dam/corporate/abbvie-com2/pdfs/joinus/2026-abv-total-rewards-resources-guide-jan-publish.pdf for *
  • Added Disallow: /content/dam/corporate/abbvie-com2/pdfs/joinus/acceptable-documents.pdf for *
  • Added Disallow: /content/dam/corporate/abbvie-com2/pdfs/joinus/total-rewards-presentation.pdf for *
  • … 10 more change(s)
view diff → snapshot
Figma Community figma.com modified +6 −0
  • 🤖 AI Added Allow: /id-id/resource-library/alat-desain-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/resource-library/strumenti-design-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/resource-library/ai-design-tools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /nl-nl/resource-library/ai-ontwerptools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pl-pl/resource-library/narzedzia-projektowania-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /th-th/resource-library/ai-design-tools/$ for CHATGPT-User
  • … 24 more change(s)
view diff → snapshot