NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2658snapshots
1487changes detected
242this week
All changes 🤖 AI-crawler changes Showing initial archives
Le Monde lemonde.fr modified +39 −0
  • Blocked user-agent ‘CohereBot’ entirely
  • Blocked user-agent ‘CragCrawler’ entirely
  • Blocked user-agent ‘GeistHaus-PageFetcher’ entirely
  • Blocked user-agent ‘GoogleAgent-URLContext’ entirely
  • Blocked user-agent ‘Kimi-SearchBot’ entirely
  • Blocked user-agent ‘Kimi-User’ entirely
  • … 7 more change(s)
view diff → snapshot
Figma Community figma.com modified +13 −0
  • 🤖 AI Added Allow: /blog/figma-is-now-iso-42001-certified/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/appi/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/australia-privacy-act/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/australia-prudential-regulation/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/eu-data-act/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
University of Cape Town uct.ac.za modified +24 −2
  • Added Disallow: /*/search for Bingbot
  • Added Disallow: /*/search$ for Bingbot
  • Added Disallow: /*/search/ for Bingbot
  • Added Disallow: /*/search? for Bingbot
  • Added Disallow: /*?*search= for Bingbot
  • Added Disallow: /admin/ for Bingbot
  • … 20 more change(s)
view diff → snapshot
Snapchat snapchat.com modified +3 −0
  • Added Disallow: /entertainment/ for *
  • Added Disallow: /entertainment/ for Googlebot
  • Added Disallow: /entertainment/ for Meta-WebIndexer
view diff → snapshot
S&P Global spglobal.com modified +0 −5
  • Removed Disallow: /automotive-insights/en-test/* for *
  • Removed Disallow: /engineering/en/search for *
  • Removed Disallow: /mobility/en/search for *
  • Removed sitemap https://www.spglobal.com/automotive-insights/en/sitemap.xml
  • Removed sitemap https://www.spglobal.com/mobility/en/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/how-to-design-agentic-tools-for-work/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/karri-saarinens-10-rules-for-crafting-products-that-stand-out/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-to-design-agentic-tools-for-work/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/karri-saarinens-10-rules-for-crafting-products-that-stand-out/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/how-to-design-agentic-tools-for-work/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/karri-saarinens-10-rules-for-crafting-products-that-stand-out/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Fortra fortra.com modified +45 −0
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 5 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Indeed indeed.com modified +30 −177
  • 🤖 AI Added rules for AI crawler AI2Bot (Allen Institute for AI)
  • 🤖 AI Added rules for AI crawler Diffbot (Diffbot)
  • Added rules for user-agent ‘Google-Agent’
  • Added rules for user-agent ‘Google-CloudVertexBot’
  • Added rules for user-agent ‘Grok-DeepSearch’
  • Added rules for user-agent ‘GrokBot’
  • … 17 more change(s)
view diff → snapshot