NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2530snapshots
1357changes detected
230this week
All changes 🤖 AI-crawler changes Hiding initial archives
Commonwealth Bank commbank.com.au modified +89 −104
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • … 45 more change(s)
view diff → snapshot
Weibo weibo.com modified +35 −15
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘ChatGLM’ entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • Blocked user-agent ‘DeepSeek’ entirely
  • Blocked user-agent ‘DeepSeek Chat’ entirely
  • Blocked user-agent ‘DeepSeekBot’ entirely
  • … 16 more change(s)
view diff → snapshot
Iceland (Government.is) government.is modified +55 −0
  • Added Disallow: /404$ for *
  • Added Disallow: /404/$ for *
  • Added Disallow: /404/? for *
  • Added Disallow: /404? for *
  • Added Disallow: /library$ for *
  • Added Disallow: /library/$ for *
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Iceland (Government.is) government.is modified +0 −55
  • Removed Disallow: /404$ for *
  • Removed Disallow: /404/$ for *
  • Removed Disallow: /404/? for *
  • Removed Disallow: /404? for *
  • Removed Disallow: /library$ for *
  • Removed Disallow: /library/$ for *
  • … 45 more change(s)
view diff → snapshot
University of Pennsylvania upenn.edu modified +6 −0
  • Added Allow: /core/*.avif for *
  • Added Allow: /core/*.webp for *
  • Added Allow: /profiles/*.avif for *
  • Added Allow: /profiles/*.webp for *
  • Added Disallow: /index.php/search? for *
  • Added Disallow: /search? for *
view diff → snapshot
ChatGPT chatgpt.com modified +3 −0
  • Added Allow: /acquisition-landing-pages-sitemap.xml for *
  • Added Allow: /ask/ for *
  • Added sitemap https://chatgpt.com/acquisition-landing-pages-sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Qonto qonto.com modified +0 −3
  • Removed Disallow: /en-at/swift-codes/* for *
  • Removed Disallow: /en-be/swift-codes/* for *
  • Removed Disallow: /en-pt/swift-codes/* for *
view diff → snapshot
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /blog/code-craft-and-the-making-of-nested-folders/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/ai-creative-process/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/ai-tools-for-graphic-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/claude-skill-design-system/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/design-operations/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/user-centered-design-questions/$ for CHATGPT-User
  • … 22 more change(s)
view diff → snapshot
Tanium tanium.com modified +24 −1
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Google-Extended (Google)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
  • … 4 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot