NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2647snapshots
1476changes detected
231this week
All changes 🤖 AI-crawler changes Hiding initial archives
Slack slack.com modified +0 −3
  • Removed Allow: /*?*&geocode=* for *
  • Removed Allow: /*?geocode=* for *
  • Removed Disallow: /*?* for *
view diff → snapshot
Figma Community figma.com modified +6 −3
  • 🤖 AI Added Allow: /blog/how-season-5-merch-came-to-life/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • … 30 more change(s)
view diff → snapshot
DSV dsv.com removed +0 −34
  • robots.txt is no longer served (HTTP 404)
Gilead Sciences gilead.com modified +2 −0
  • Added Disallow: /-/media/gileadspain/pdfs/about/transparency/2025/gilead-sciences-spain-efpia-methodological-note-2025.pdf for *
  • Added Disallow: /-/media/gileadspain/pdfs/about/transparency/2025/gilead-sciences-spain-efpia-report-2025.pdf for *
view diff → snapshot
Qonto qonto.com modified +2 −490
  • Removed all rules for user-agent ‘AITCSRobot/1.1’
  • Removed all rules for user-agent ‘Alexibot’
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • Removed all rules for user-agent ‘Aqua_Products’
  • Removed all rules for user-agent ‘Arachnophilia’
  • Removed all rules for user-agent ‘ASpider/0.09’
  • … 45 more change(s)
view diff → snapshot
Al Jazeera aljazeera.com modified +1 −1
  • Added Disallow: /*/liveblog/*?*update=* for *
  • Removed Disallow: /*/liveblog/2026/6/*?*update=* for *
view diff → snapshot
Figma Community figma.com modified +30 −0
  • 🤖 AI Added Allow: /de-de/blog/4-new-ways-to-go-from-idea-to-product-with-ai-tools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/agent-custom-tools-context-skills/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/code-on-the-figma-canvas/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/config-2026-recap/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/figma-make-now-on-your-local-code/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/introducing-figma-motion/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Kimberly-Clark kimberly-clark.com modified +90 −20
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • Added rules for user-agent ‘Applebot’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • … 22 more change(s)
view diff → snapshot
ACS Group grupoacs.com modified +2 −1
  • Added Disallow: /accionistas-e-inversores-nuevo/* for *
  • Added Disallow: /shareholders-investors-new/* for *
  • Removed Disallow: (empty) for *
view diff → snapshot
University of Cape Town uct.ac.za modified +2 −2
  • Unblocked user-agent ‘Bingbot’
  • Added Disallow: (empty) for Bingbot
  • 🤖 AI Unblocked OAI-SearchBot (OpenAI)
  • 🤖 AI Added Disallow: (empty) for OAI-SearchBot
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Honeywell International honeywell.com modified +27 −62
  • Added Disallow: /bin/edam/getfileservlet for *
  • Added Disallow: /pif/ for *
  • Added Disallow: /us/en/legal/cookie-notice?ref=* for *
  • Removed Disallow: /br/pt?utm* for *
  • Removed Disallow: /content/dam/honeywellbt/en/documents/downloads/hon-HoneywellForge_ForBuildings_Brochure_042020.pdf for *
  • Removed Disallow: /content/dam/honeywellbt/en/documents/downloads/honeywell-forge/Honeywell_Forge_Terms.pdf for *
  • … 45 more change(s)
view diff → snapshot