NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
All changes 🤖 AI-crawler changes Hiding initial archives
Figma Community figma.com modified +15 −0
  • 🤖 AI Added Allow: /de-de/blog/improving-performance-in-the-layers-panel/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/improving-performance-in-the-layers-panel/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/improving-performance-in-the-layers-panel/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/improving-performance-in-the-layers-panel/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/improving-performance-in-the-layers-panel/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/improving-performance-in-the-layers-panel/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Slate slate.com modified +10 −0
  • Blocked user-agent ‘bedrockbot’ entirely
  • Blocked user-agent ‘DeepSeekBot’ entirely
  • Blocked user-agent ‘Doubaobot’ entirely
  • Blocked user-agent ‘ERNIEBot’ entirely
  • Blocked user-agent ‘KimiCrawler’ entirely
  • Blocked user-agent ‘micro-crawl’ entirely
  • … 4 more change(s)
view diff → snapshot
Snapchat snapchat.com modified +42 −23
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler Perplexity-User (Perplexity)
  • Removed all rules for user-agent ‘Perplexity‑User’
  • Removed Disallow: /entertainment/ for *
  • 🤖 AI Unblocked ChatGPT-User (OpenAI)
  • … 17 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Salesforce salesforce.com modified +2 −0
  • Added Disallow: /*/wp-content/uploads/sites/*/documents/legal/no-index/*.pdf$ for AdsBot-Google
  • Added Disallow: /*/wp-content/uploads/sites/*/partners/*.pdf$ for AdsBot-Google
view diff → snapshot
Abbott Laboratories abbott.com modified +48 −0
  • Added Disallow: /de-de/errorpages for *
  • Added Disallow: /en-anz/errorpages for *
  • Added Disallow: /en-ca/errorpages for *
  • Added Disallow: /en-ie/errorpages for *
  • Added Disallow: /en-in/errorpages for *
  • Added Disallow: /en-my/errorpages for *
  • … 32 more change(s)
view diff → snapshot
The Economist economist.com modified +49 −21
  • 🤖 AI Blocked OAI-SearchBot (OpenAI) entirely
  • 🤖 AI Added Allow: /pro/ for ChatGPT-User
  • 🤖 AI Added Allow: /pro/ for Claude-Web
  • 🤖 AI Added Allow: /pro/ for Perplexity-User
  • 🤖 AI Added Allow: /pro/ for PerplexityBot
view diff → snapshot
BBC bbc.com modified +3 −2
  • Added sitemap https://www.bbc.com/sitemaps/world-service-homepages.xml
view diff → snapshot
Barracuda barracuda.com removed +0 −232
  • robots.txt is no longer served (HTTP 404)
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Figma Community figma.com modified +43 −0
  • 🤖 AI Added Allow: /blog/sightlines-issue-no1/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/figma-make-general-availability/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/future-states/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/gpt-5-6-is-now-available-in-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/figma-make-general-availability/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/future-states/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Europe PMC europepmc.org modified +98 −7
  • Added rules for user-agent ‘baiduspider’
  • Added rules for user-agent ‘bingbot’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
  • … 21 more change(s)
view diff → snapshot