NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
All changes 🤖 AI-crawler changes Hiding initial archives
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/building-accessibility-into-a-canvas-based-product/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/building-accessibility-into-a-canvas-based-product/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/building-accessibility-into-a-canvas-based-product/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/got-skills-make-the-figma-agent-a-better-collaborator/$ for OAI-SearchBot
  • … 2 more change(s)
view diff → snapshot
Microsoft microsoft.com modified +9 −1
  • Added Allow: /en/microsoft-365/free-office-online-for-the-web for *
  • Added Allow: /en/microsoft-365/onenote/digital-note-taking-app for *
  • Added Allow: /en/microsoft-365/outlook/web-email-login-for-outlook for *
  • Added Allow: /en/microsoft-365/planner/microsoft-planner for *
  • Added Allow: /en/microsoft-365/products-apps-services for *
  • Added Allow: /en/microsoft-365/publisher for *
  • … 2 more change(s)
view diff → snapshot
Analog Devices analog.com modified +4 −0
  • Added Disallow: /cn/crsearch/ for *
  • Added Disallow: /en/crsearch/ for *
  • Added Disallow: /jp/crsearch/ for *
  • Added Disallow: /sitecore/ for *
view diff → snapshot
Volvo Cars volvocars.com modified +5 −2
  • Added rules for user-agent ‘YisouSpider’
  • 🤖 AI Added Disallow: /*/build/* for Amazonbot
  • 🤖 AI Added Disallow: /*/order-request/* for Amazonbot
  • 🤖 AI Removed Disallow: /*/build/*?* for Amazonbot
  • 🤖 AI Removed Disallow: /*/order* for Amazonbot
  • 🤖 AI Added Disallow: /*/build/* for anthropic-ai
  • … 45 more change(s)
view diff → snapshot
Le Monde lemonde.fr modified +39 −0
  • Blocked user-agent ‘CohereBot’ entirely
  • Blocked user-agent ‘CragCrawler’ entirely
  • Blocked user-agent ‘GeistHaus-PageFetcher’ entirely
  • Blocked user-agent ‘GoogleAgent-URLContext’ entirely
  • Blocked user-agent ‘Kimi-SearchBot’ entirely
  • Blocked user-agent ‘Kimi-User’ entirely
  • … 7 more change(s)
view diff → snapshot
Figma Community figma.com modified +13 −0
  • 🤖 AI Added Allow: /blog/figma-is-now-iso-42001-certified/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/appi/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/australia-privacy-act/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/australia-prudential-regulation/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/eu-data-act/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
University of Cape Town uct.ac.za modified +24 −2
  • Added Disallow: /*/search for Bingbot
  • Added Disallow: /*/search$ for Bingbot
  • Added Disallow: /*/search/ for Bingbot
  • Added Disallow: /*/search? for Bingbot
  • Added Disallow: /*?*search= for Bingbot
  • Added Disallow: /admin/ for Bingbot
  • … 20 more change(s)
view diff → snapshot
Snapchat snapchat.com modified +3 −0
  • Added Disallow: /entertainment/ for *
  • Added Disallow: /entertainment/ for Googlebot
  • Added Disallow: /entertainment/ for Meta-WebIndexer
view diff → snapshot
S&P Global spglobal.com modified +0 −5
  • Removed Disallow: /automotive-insights/en-test/* for *
  • Removed Disallow: /engineering/en/search for *
  • Removed Disallow: /mobility/en/search for *
  • Removed sitemap https://www.spglobal.com/automotive-insights/en/sitemap.xml
  • Removed sitemap https://www.spglobal.com/mobility/en/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot