NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2575snapshots
1402changes detected
229this week
All changes 🤖 AI-crawler changes Showing initial archives
Figma Community figma.com modified +13 −0
  • 🤖 AI Added Allow: /blog/figma-is-now-iso-42001-certified/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/appi/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/australia-privacy-act/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/australia-prudential-regulation/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/eu-data-act/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
University of Cape Town uct.ac.za modified +24 −2
  • Added Disallow: /*/search for Bingbot
  • Added Disallow: /*/search$ for Bingbot
  • Added Disallow: /*/search/ for Bingbot
  • Added Disallow: /*/search? for Bingbot
  • Added Disallow: /*?*search= for Bingbot
  • Added Disallow: /admin/ for Bingbot
  • … 20 more change(s)
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/how-to-design-agentic-tools-for-work/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/karri-saarinens-10-rules-for-crafting-products-that-stand-out/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-to-design-agentic-tools-for-work/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/karri-saarinens-10-rules-for-crafting-products-that-stand-out/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/how-to-design-agentic-tools-for-work/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/karri-saarinens-10-rules-for-crafting-products-that-stand-out/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Fortra fortra.com modified +45 −0
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 5 more change(s)
view diff → snapshot
Indeed indeed.com modified +30 −177
  • 🤖 AI Added rules for AI crawler AI2Bot (Allen Institute for AI)
  • 🤖 AI Added rules for AI crawler Diffbot (Diffbot)
  • Added rules for user-agent ‘Google-Agent’
  • Added rules for user-agent ‘Google-CloudVertexBot’
  • Added rules for user-agent ‘Grok-DeepSearch’
  • Added rules for user-agent ‘GrokBot’
  • … 17 more change(s)
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/bill-atkinsons-10-rules-for-making-interfaces-more-human/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/resource-library/ki-website-beispiele/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/resource-library/ejemplos-sitios-web-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/andrew-boz-bosworths-10-rules-for-navigating-the-next-design-paradigm/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/bill-atkinsons-10-rules-for-making-interfaces-more-human/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/resource-library/ejemplos-sitios-web-de-ia/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
LinkedIn linkedin.com modified +120 −1
  • 🤖 AI Unblocked Claude-SearchBot (Anthropic)
  • 🤖 AI Added Allow: /business/learning/blog* for Claude-SearchBot
  • 🤖 AI Added Allow: /business/sales/blog* for Claude-SearchBot
  • 🤖 AI Added Allow: /help/ for Claude-SearchBot
  • 🤖 AI Added Allow: /psettings/guest-controls* for Claude-SearchBot
  • 🤖 AI Added Allow: /psettings/guest-controls/retargeting-opt-out* for Claude-SearchBot
  • … 45 more change(s)
view diff → snapshot
Flickr flickr.com modified +5 −0
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
view diff → snapshot
Figma Community figma.com modified +6 −3
  • 🤖 AI Added Allow: /blog/how-season-5-merch-came-to-life/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • … 30 more change(s)
view diff → snapshot
Qonto qonto.com modified +2 −490
  • Removed all rules for user-agent ‘AITCSRobot/1.1’
  • Removed all rules for user-agent ‘Alexibot’
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • Removed all rules for user-agent ‘Aqua_Products’
  • Removed all rules for user-agent ‘Arachnophilia’
  • Removed all rules for user-agent ‘ASpider/0.09’
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +30 −0
  • 🤖 AI Added Allow: /de-de/blog/4-new-ways-to-go-from-idea-to-product-with-ai-tools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/agent-custom-tools-context-skills/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/code-on-the-figma-canvas/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/config-2026-recap/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/figma-make-now-on-your-local-code/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/introducing-figma-motion/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Kimberly-Clark kimberly-clark.com modified +90 −20
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • Added rules for user-agent ‘Applebot’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • … 22 more change(s)
view diff → snapshot
University of Cape Town uct.ac.za modified +2 −2
  • Unblocked user-agent ‘Bingbot’
  • Added Disallow: (empty) for Bingbot
  • 🤖 AI Unblocked OAI-SearchBot (OpenAI)
  • 🤖 AI Added Disallow: (empty) for OAI-SearchBot
view diff → snapshot
Corriere della Sera corriere.it modified +10 −0
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘Diffbot-User’ entirely
  • Blocked user-agent ‘Scrapy’ entirely
  • 🤖 AI Blocked Timpibot (Timpi) entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
view diff → snapshot
Corriere della Sera corriere.it modified +0 −10
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • Removed all rules for user-agent ‘Diffbot-User’
  • Removed all rules for user-agent ‘Scrapy’
  • 🤖 AI Unblocked Timpibot (Timpi) - rules removed
  • 🤖 AI Unblocked YouBot (You.com) - rules removed
view diff → snapshot
Corriere della Sera corriere.it modified +10 −0
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘Diffbot-User’ entirely
  • Blocked user-agent ‘Scrapy’ entirely
  • 🤖 AI Blocked Timpibot (Timpi) entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
view diff → snapshot
Corriere della Sera corriere.it modified +0 −10
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • Removed all rules for user-agent ‘Diffbot-User’
  • Removed all rules for user-agent ‘Scrapy’
  • 🤖 AI Unblocked Timpibot (Timpi) - rules removed
  • 🤖 AI Unblocked YouBot (You.com) - rules removed
view diff → snapshot
Grupo Bimbo grupobimbo.com modified +120 −0
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Google-Extended (Google)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • … 3 more change(s)
view diff → snapshot
Corriere della Sera corriere.it modified +10 −0
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘Diffbot-User’ entirely
  • Blocked user-agent ‘Scrapy’ entirely
  • 🤖 AI Blocked Timpibot (Timpi) entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
view diff → snapshot
DataDome datadome.co modified +37 −0
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Added rules for user-agent ‘Bingbot’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler cohere-ai (Cohere)
  • … 4 more change(s)
view diff → snapshot
Corriere della Sera corriere.it modified +0 −10
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • Removed all rules for user-agent ‘Diffbot-User’
  • Removed all rules for user-agent ‘Scrapy’
  • 🤖 AI Unblocked Timpibot (Timpi) - rules removed
  • 🤖 AI Unblocked YouBot (You.com) - rules removed
view diff → snapshot
Indeed indeed.com modified +6 −52
  • Added Disallow: /*&serpstart= for *
  • Added Disallow: /*/vaclk for *
  • Removed Allow: /hire/*?*isid= for *
  • Removed Allow: /personeel/*?*isid= for *
  • Removed Allow: /reclutamiento/*?*isid= for *
  • Removed Allow: /recruiting/*?*isid= for *
  • … 45 more change(s)
view diff → snapshot
SoundCloud soundcloud.com modified +29 −60
  • Blocked user-agent ‘Brightbot’ entirely
  • 🤖 AI Blocked omgili (Webz.io) entirely
  • Blocked user-agent ‘omgilibot’ entirely
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • … 45 more change(s)
view diff → snapshot
Corriere della Sera corriere.it modified +10 −0
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘Diffbot-User’ entirely
  • Blocked user-agent ‘Scrapy’ entirely
  • 🤖 AI Blocked Timpibot (Timpi) entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
view diff → snapshot
SoundCloud soundcloud.com modified +60 −29
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Blocked cohere-ai (Cohere) entirely
  • … 45 more change(s)
view diff → snapshot
BuzzFeed buzzfeed.com modified +1 −0
  • 🤖 AI Added Allow: /rss-feeds/feeds/bf-amazon-ai-grounding.extended.xml for Amazonbot
view diff → snapshot
Figma Community figma.com modified +94 −0
  • 🤖 AI Added Allow: /de-de/resource-library/ki-coding-tools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/resource-library/was-ist-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/resource-library/herramientas-programacion-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/resource-library/que-es-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/2025-shifting-roles-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/4-ways-for-design-teams-to-chart-new-territory-with-figma-make/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
El País elpais.com modified +14 −0
  • 🤖 AI Blocked Claude-SearchBot (Anthropic) entirely
  • Blocked user-agent ‘Comet’ entirely
  • 🤖 AI Blocked Perplexity-User (Perplexity) entirely
  • 🤖 AI Blocked PerplexityBot (Perplexity) entirely
view diff → snapshot