NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2575snapshots
1402changes detected
229this week
All changes 🤖 AI-crawler changes Showing initial archives
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/white-papers/pfpt-us-wp-why-ai-security-is-not-an-endpoint-network-or-firewall-problem.pdf for *
view diff → snapshot
Etsy etsy.com modified +60 −0
  • Added Disallow: /appeals/ for *
  • Added Disallow: /at/appeals/ for *
  • Added Disallow: /at/r/group/ for *
  • Added Disallow: /au/appeals/ for *
  • Added Disallow: /au/r/group/ for *
  • Added Disallow: /be/appeals/ for *
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for PerplexityBot
view diff → snapshot
Scale AI scale.com modified +0 −1
  • Removed sitemap https://scale.com/enterprise/prebuilt-applications/sitemap.xml
view diff → snapshot
Ford Motor ford.com modified +10 −6
  • Added sitemap https://www.ford.com/inventory/llms.txt
  • Added sitemap https://www.ford.com/llms.txt
  • Removed sitemap https://www.ford.com/href-sitemap-en-us.xml.
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/white-papers/pfpt-us-wp-agent-integrity-framework.pdf for *
view diff → snapshot
Twitch twitch.tv modified +4 −2
  • Added Disallow: /gl$ for *
  • Added Disallow: /gl/ for *
  • Added Disallow: /sq$ for *
  • Added Disallow: /sq/ for *
  • Removed Disallow: /gl for *
  • Removed Disallow: /sq for *
view diff → snapshot
Yandex yandex.ru modified +6 −0
  • Added Disallow: /realty*?*text= for *
  • Added Disallow: /realty/search for *
  • Added Disallow: /realty*?*text= for Bingbot
  • Added Disallow: /realty/search for Bingbot
  • Added Disallow: /realty*?*text= for Googlebot
  • Added Disallow: /realty/search for Googlebot
view diff → snapshot
Nubank nubank.com.br modified +0 −1392
  • Unblocked user-agent ‘*’
  • Removed Disallow: /ajuda-e-seguranca for *
  • Removed Disallow: /ajuda-e-seguranca/central-de-protecao for *
  • Removed Disallow: /ajuda-e-seguranca/rotativo-e-parcelamento for *
  • Removed Disallow: /ajuda-e-seguranca/selo-de-prevencao-a-fraude for *
  • Removed Disallow: /buscar for *
  • … 45 more change(s)
view diff → snapshot
MetLife metlife.com modified +17 −8
  • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prodtenants.adobecqms.net/* for *
  • Removed Disallow: https://metlife-qa1-65.adobecqms.net/* for *
  • … 18 more change(s)
view diff → snapshot
Morgan Stanley morganstanley.com modified +46 −6
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • … 8 more change(s)
view diff → snapshot
Figma Community figma.com modified +24 −0
  • 🤖 AI Added Allow: /de-de/blog/how-decagon-uses-ai-for-design-system-saturation/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/sightlines-issue-no1/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/steal-this-template-bring-a-user-persona-to-life-with-figma-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/workflow-lab-deploying-designs-directly-with-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-decagon-uses-ai-for-design-system-saturation/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/sightlines-issue-no1/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot