NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Hiding initial archives
Twitch twitch.tv modified +4 −2
  • Added Disallow: /gl$ for *
  • Added Disallow: /gl/ for *
  • Added Disallow: /sq$ for *
  • Added Disallow: /sq/ for *
  • Removed Disallow: /gl for *
  • Removed Disallow: /sq for *
view diff → snapshot
Yandex yandex.ru modified +6 −0
  • Added Disallow: /realty*?*text= for *
  • Added Disallow: /realty/search for *
  • Added Disallow: /realty*?*text= for Bingbot
  • Added Disallow: /realty/search for Bingbot
  • Added Disallow: /realty*?*text= for Googlebot
  • Added Disallow: /realty/search for Googlebot
view diff → snapshot
Nubank nubank.com.br modified +0 −1392
  • Unblocked user-agent ‘*’
  • Removed Disallow: /ajuda-e-seguranca for *
  • Removed Disallow: /ajuda-e-seguranca/central-de-protecao for *
  • Removed Disallow: /ajuda-e-seguranca/rotativo-e-parcelamento for *
  • Removed Disallow: /ajuda-e-seguranca/selo-de-prevencao-a-fraude for *
  • Removed Disallow: /buscar for *
  • … 45 more change(s)
view diff → snapshot
MetLife metlife.com modified +17 −8
  • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prodtenants.adobecqms.net/* for *
  • Removed Disallow: https://metlife-qa1-65.adobecqms.net/* for *
  • … 18 more change(s)
view diff → snapshot
Morgan Stanley morganstanley.com modified +46 −6
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • … 8 more change(s)
view diff → snapshot
Figma Community figma.com modified +24 −0
  • 🤖 AI Added Allow: /de-de/blog/how-decagon-uses-ai-for-design-system-saturation/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/sightlines-issue-no1/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/steal-this-template-bring-a-user-persona-to-life-with-figma-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/workflow-lab-deploying-designs-directly-with-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-decagon-uses-ai-for-design-system-saturation/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/sightlines-issue-no1/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
MetLife metlife.com modified +8 −2
  • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
  • Added Disallow: https://metlife-qa1-65.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Nubank nubank.com.br modified +1392 −0
  • Added Disallow: /ajuda-e-seguranca for *
  • Added Disallow: /ajuda-e-seguranca/central-de-protecao for *
  • Added Disallow: /ajuda-e-seguranca/rotativo-e-parcelamento for *
  • Added Disallow: /ajuda-e-seguranca/selo-de-prevencao-a-fraude for *
  • Added Disallow: /buscar for *
  • Added Disallow: /chatgpt-go for *
  • … 45 more change(s)
view diff → snapshot