NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2601snapshots
1428changes detected
255this week
All changes 🤖 AI-crawler changes Showing initial archives
MetLife metlife.com modified +2 −8
  • Added Disallow: /investments$ for *
  • Added Disallow: /investments/ for *
  • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Vinci vinci.com modified +6 −0
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
view diff → snapshot
Ahold Delhaize aholddelhaize.com modified +2 −16
  • Removed Allow: / for *
  • Removed Disallow: (empty) for *
  • Removed Disallow: /*.json$ for *
  • Removed Disallow: /*.xml$ for *
  • Removed Disallow: /_nuxt/ for *
  • Removed Disallow: /admin/ for *
  • … 8 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Figma Community figma.com modified +11 −0
  • 🤖 AI Added Allow: /de-de/loesungen/ki-barrierefreiheits-checker/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/solutions/verificateur-accessibilite-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/2026-ai-report/$ for CHATGPT-User
  • … 38 more change(s)
view diff → snapshot
Roblox roblox.com modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/misc/pfpt-us-rt-2026-ai-era-ransomware.pdf for *
view diff → snapshot
SoundCloud soundcloud.com modified +29 −60
  • Blocked user-agent ‘Brightbot’ entirely
  • 🤖 AI Blocked omgili (Webz.io) entirely
  • Blocked user-agent ‘omgilibot’ entirely
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • … 45 more change(s)
view diff → snapshot
AGCO agcocorp.com modified +1 −0
  • Added sitemap https://www.agcocorp.com/us/en/home/compliance-center/tax-strategy/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot