NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2601snapshots
1428changes detected
255this week
All changes 🤖 AI-crawler changes Hiding initial archives
Johnson & Johnson jnj.com modified +0 −8
  • Removed Allow: /media-center/media-contacts for *
  • Removed Disallow: /autodiscover/autodiscover.xml for *
  • Removed Disallow: /innovation/8-innovative-office-perks-designed-to-make-you-happier-at-work for *
  • Removed Disallow: /innovation/touring-the-johnson-johnson-institute-irvine for *
  • Removed Disallow: /media-center/* for *
  • Removed Disallow: /news/all/Investigational-HIV-Vaccine-Regimen-Showc%20s-Encouraging-Results-in-Non-Human-Primates for *
  • … 2 more change(s)
view diff → snapshot
Figma Community figma.com modified +31 −0
  • 🤖 AI Added Allow: /de-de/blog/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/jen-dunnam-lead-design-teams/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/sightlines/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-footer-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-hintergrundentferner/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Roblox roblox.com modified +61 −0
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • Blocked user-agent ‘CloudflareBrowserRenderingCrawler’ entirely
  • … 4 more change(s)
view diff → snapshot
The Washington Post washingtonpost.com modified +1 −3
  • 🤖 AI Unblocked PerplexityBot (Perplexity)
  • 🤖 AI Added Allow: / for PerplexityBot
  • 🤖 AI Removed Allow: /advertising/ for PerplexityBot
  • 🤖 AI Removed Allow: /creativegroup/ for PerplexityBot
view diff → snapshot
SoundCloud soundcloud.com modified +60 −29
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Blocked cohere-ai (Cohere) entirely
  • … 45 more change(s)
view diff → snapshot
Tractor Supply tractorsupply.com modified +12 −0
  • Added Disallow: /gtwy/api/accountservice/orchestration/accounts/change/password for *
  • Added Disallow: /gtwy/api/accountservice/orchestration/accounts/forgot/password for *
  • Added Disallow: /gtwy/api/accountservice/orchestration/accounts/lookup for *
  • Added Disallow: /gtwy/api/accountservice/orchestration/accounts/update for *
  • Added Disallow: /gtwy/api/accountservice/orchestration/accounts/users/ncenroll for *
  • Added Disallow: /gtwy/api/accountservice/orchestration/accounts/verifyTicket for *
  • … 6 more change(s)
view diff → snapshot
Owens Corning owenscorning.com modified +1 −32
  • Removed all rules for user-agent ‘*’
  • Removed sitemap https://www.owenscorning.com/sitemap-1.xml
  • Removed sitemap https://www.owenscorning.com/sitemap.xml
view diff → snapshot
Societe Generale societegenerale.com removed +0 −63
  • robots.txt is no longer served (HTTP 404)
Barracuda barracuda.com created +28 −28
  • Added Disallow: /70hkWk5EPQLRDbV9Q7189DJB_y14lTx9BS0qLGJVSZA=.html for *
  • Removed Disallow: /2QGvuWDJzNTl_zQtUa-kCYxc421SSSioP4rLS4V3Tic=.html for *
  • Added Disallow: /70hkWk5EPQLRDbV9Q7189DJB_y14lTx9BS0qLGJVSZA=.html for DotBot
  • Removed Disallow: /2QGvuWDJzNTl_zQtUa-kCYxc421SSSioP4rLS4V3Tic=.html for DotBot
  • Added Disallow: /70hkWk5EPQLRDbV9Q7189DJB_y14lTx9BS0qLGJVSZA=.html for sistrix
  • Removed Disallow: /2QGvuWDJzNTl_zQtUa-kCYxc421SSSioP4rLS4V3Tic=.html for sistrix
  • … 2 more change(s)
view diff → snapshot
Owens Corning owenscorning.com modified +32 −1
  • Added rules for user-agent ‘*’
  • Added sitemap https://www.owenscorning.com/sitemap-1.xml
  • Added sitemap https://www.owenscorning.com/sitemap.xml
view diff → snapshot
Barracuda barracuda.com removed +0 −232
  • robots.txt is no longer served (HTTP 404)
L'Oreal loreal.com modified +3 −0
  • Added sitemap https://www.loreal.com/sitemap-image.xml
  • Added sitemap https://www.loreal.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot