NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
All changes 🤖 AI-crawler changes Hiding initial archives
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Figma Community figma.com modified +11 −0
  • 🤖 AI Added Allow: /de-de/loesungen/ki-barrierefreiheits-checker/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/solutions/verificateur-accessibilite-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/2026-ai-report/$ for CHATGPT-User
  • … 38 more change(s)
view diff → snapshot
Roblox roblox.com modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/misc/pfpt-us-rt-2026-ai-era-ransomware.pdf for *
view diff → snapshot
SoundCloud soundcloud.com modified +29 −60
  • Blocked user-agent ‘Brightbot’ entirely
  • 🤖 AI Blocked omgili (Webz.io) entirely
  • Blocked user-agent ‘omgilibot’ entirely
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • … 45 more change(s)
view diff → snapshot
AGCO agcocorp.com modified +1 −0
  • Added sitemap https://www.agcocorp.com/us/en/home/compliance-center/tax-strategy/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Nubank nubank.com.br modified +0 −12
  • Removed Disallow: /home for *
  • Removed Disallow: /home2 for *
  • Removed Disallow: /home for DatoCmsSearchBot
  • Removed Disallow: /home2 for DatoCmsSearchBot
view diff → snapshot
Johnson & Johnson jnj.com modified +0 −8
  • Removed Allow: /media-center/media-contacts for *
  • Removed Disallow: /autodiscover/autodiscover.xml for *
  • Removed Disallow: /innovation/8-innovative-office-perks-designed-to-make-you-happier-at-work for *
  • Removed Disallow: /innovation/touring-the-johnson-johnson-institute-irvine for *
  • Removed Disallow: /media-center/* for *
  • Removed Disallow: /news/all/Investigational-HIV-Vaccine-Regimen-Showc%20s-Encouraging-Results-in-Non-Human-Primates for *
  • … 2 more change(s)
view diff → snapshot
Figma Community figma.com modified +31 −0
  • 🤖 AI Added Allow: /de-de/blog/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/jen-dunnam-lead-design-teams/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/sightlines/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-footer-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-hintergrundentferner/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Roblox roblox.com modified +61 −0
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • Blocked user-agent ‘CloudflareBrowserRenderingCrawler’ entirely
  • … 4 more change(s)
view diff → snapshot
The Washington Post washingtonpost.com modified +1 −3
  • 🤖 AI Unblocked PerplexityBot (Perplexity)
  • 🤖 AI Added Allow: / for PerplexityBot
  • 🤖 AI Removed Allow: /advertising/ for PerplexityBot
  • 🤖 AI Removed Allow: /creativegroup/ for PerplexityBot
view diff → snapshot