NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2601snapshots
1428changes detected
255this week
All changes 🤖 AI-crawler changes Hiding initial archives
L'Oreal loreal.com modified +0 −3
  • Removed sitemap https://www.loreal.com/sitemap-image.xml
  • Removed sitemap https://www.loreal.com/sitemap.xml
view diff → snapshot
Roblox roblox.com modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
General Mills generalmills.com modified +2 −9
  • Added Allow: / for *
  • Removed Disallow: /*.aspx for *
  • Removed Disallow: /*p=1 for *
  • Removed Disallow: /.well-known/apple-app-site-association for *
  • Removed Disallow: /apple-app-site-association for *
  • Removed Disallow: /search for *
  • … 4 more change(s)
view diff → snapshot
Renault Group renaultgroup.com modified +0 −2
  • Removed Disallow: https://back.renaultgroup.com/wp-admin/ for *
  • Removed Disallow: https://back.renaultgroup.com/wp-json/ for *
view diff → snapshot
Waldo waldo.fyi modified +9 −0
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Google-Extended (Google)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • … 1 more change(s)
view diff → snapshot
Le Figaro lefigaro.fr modified +2 −0
  • 🤖 AI Added Allow: /beaute for ChatGPT-User
  • 🤖 AI Added Allow: /beaute for OAI-SearchBot
view diff → snapshot
The Economist economist.com modified +19 −1
  • 🤖 AI Blocked Claude-SearchBot (Anthropic) entirely
  • 🤖 AI Blocked Claude-User (Anthropic) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Added Allow: /pro$ for ChatGPT-User
  • 🤖 AI Added Allow: /pro$ for Claude-Web
  • 🤖 AI Added Allow: /pro$ for OAI-SearchBot
  • … 2 more change(s)
view diff → snapshot
Figma Community figma.com modified +22 −1
  • 🤖 AI Added Allow: /de-de/resource-library/vibe-coding-beispiele/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/resource-library/ejemplos-de-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/resource-library/ejemplos-de-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/resource-library/exemples-de-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /id-id/resource-library/contoh-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/resource-library/esempi-di-vibe-coding/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Roblox roblox.com modified +61 −0
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • Blocked user-agent ‘CloudflareBrowserRenderingCrawler’ entirely
  • … 4 more change(s)
view diff → snapshot
Weibo weibo.com modified +15 −35
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • Removed all rules for user-agent ‘ChatGLM’
  • 🤖 AI Unblocked ChatGPT-User (OpenAI) - rules removed
  • Removed all rules for user-agent ‘DeepSeek’
  • Removed all rules for user-agent ‘DeepSeek Chat’
  • Removed all rules for user-agent ‘DeepSeekBot’
  • … 16 more change(s)
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/threat-reports/pfpt-us-tr-2026-ai-era-ransomware.pdf for *
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
General Mills generalmills.com modified +9 −2
  • Added Disallow: /*.aspx for *
  • Added Disallow: /*p=1 for *
  • Added Disallow: /.well-known/apple-app-site-association for *
  • Added Disallow: /apple-app-site-association for *
  • Added Disallow: /search for *
  • Added Disallow: /shared/ for *
  • … 4 more change(s)
view diff → snapshot
Barracuda barracuda.com removed +0 −232
  • robots.txt is no longer served (HTTP 404)
Weibo weibo.com modified +35 −15
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘ChatGLM’ entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • Blocked user-agent ‘DeepSeek’ entirely
  • Blocked user-agent ‘DeepSeek Chat’ entirely
  • Blocked user-agent ‘DeepSeekBot’ entirely
  • … 16 more change(s)
view diff → snapshot