NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Showing initial archives
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/bill-atkinsons-10-rules-for-making-interfaces-more-human/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/resource-library/ki-website-beispiele/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/resource-library/ejemplos-sitios-web-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/andrew-boz-bosworths-10-rules-for-navigating-the-next-design-paradigm/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/bill-atkinsons-10-rules-for-making-interfaces-more-human/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/resource-library/ejemplos-sitios-web-de-ia/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
LinkedIn linkedin.com modified +120 −1
  • 🤖 AI Unblocked Claude-SearchBot (Anthropic)
  • 🤖 AI Added Allow: /business/learning/blog* for Claude-SearchBot
  • 🤖 AI Added Allow: /business/sales/blog* for Claude-SearchBot
  • 🤖 AI Added Allow: /help/ for Claude-SearchBot
  • 🤖 AI Added Allow: /psettings/guest-controls* for Claude-SearchBot
  • 🤖 AI Added Allow: /psettings/guest-controls/retargeting-opt-out* for Claude-SearchBot
  • … 45 more change(s)
view diff → snapshot
Flickr flickr.com modified +5 −0
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
view diff → snapshot
Figma Community figma.com modified +6 −3
  • 🤖 AI Added Allow: /blog/how-season-5-merch-came-to-life/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • … 30 more change(s)
view diff → snapshot
Qonto qonto.com modified +2 −490
  • Removed all rules for user-agent ‘AITCSRobot/1.1’
  • Removed all rules for user-agent ‘Alexibot’
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • Removed all rules for user-agent ‘Aqua_Products’
  • Removed all rules for user-agent ‘Arachnophilia’
  • Removed all rules for user-agent ‘ASpider/0.09’
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +30 −0
  • 🤖 AI Added Allow: /de-de/blog/4-new-ways-to-go-from-idea-to-product-with-ai-tools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/agent-custom-tools-context-skills/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/code-on-the-figma-canvas/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/config-2026-recap/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/figma-make-now-on-your-local-code/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/introducing-figma-motion/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Kimberly-Clark kimberly-clark.com modified +90 −20
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • Added rules for user-agent ‘Applebot’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • … 22 more change(s)
view diff → snapshot
University of Cape Town uct.ac.za modified +2 −2
  • Unblocked user-agent ‘Bingbot’
  • Added Disallow: (empty) for Bingbot
  • 🤖 AI Unblocked OAI-SearchBot (OpenAI)
  • 🤖 AI Added Disallow: (empty) for OAI-SearchBot
view diff → snapshot
Corriere della Sera corriere.it modified +10 −0
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘Diffbot-User’ entirely
  • Blocked user-agent ‘Scrapy’ entirely
  • 🤖 AI Blocked Timpibot (Timpi) entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
view diff → snapshot
Corriere della Sera corriere.it modified +0 −10
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • Removed all rules for user-agent ‘Diffbot-User’
  • Removed all rules for user-agent ‘Scrapy’
  • 🤖 AI Unblocked Timpibot (Timpi) - rules removed
  • 🤖 AI Unblocked YouBot (You.com) - rules removed
view diff → snapshot
Corriere della Sera corriere.it modified +10 −0
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘Diffbot-User’ entirely
  • Blocked user-agent ‘Scrapy’ entirely
  • 🤖 AI Blocked Timpibot (Timpi) entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
view diff → snapshot
Corriere della Sera corriere.it modified +0 −10
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • Removed all rules for user-agent ‘Diffbot-User’
  • Removed all rules for user-agent ‘Scrapy’
  • 🤖 AI Unblocked Timpibot (Timpi) - rules removed
  • 🤖 AI Unblocked YouBot (You.com) - rules removed
view diff → snapshot
Grupo Bimbo grupobimbo.com modified +120 −0
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Google-Extended (Google)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • … 3 more change(s)
view diff → snapshot
Corriere della Sera corriere.it modified +10 −0
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘Diffbot-User’ entirely
  • Blocked user-agent ‘Scrapy’ entirely
  • 🤖 AI Blocked Timpibot (Timpi) entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
view diff → snapshot
DataDome datadome.co modified +37 −0
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Added rules for user-agent ‘Bingbot’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler cohere-ai (Cohere)
  • … 4 more change(s)
view diff → snapshot
Corriere della Sera corriere.it modified +0 −10
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • Removed all rules for user-agent ‘Diffbot-User’
  • Removed all rules for user-agent ‘Scrapy’
  • 🤖 AI Unblocked Timpibot (Timpi) - rules removed
  • 🤖 AI Unblocked YouBot (You.com) - rules removed
view diff → snapshot
Indeed indeed.com modified +6 −52
  • Added Disallow: /*&serpstart= for *
  • Added Disallow: /*/vaclk for *
  • Removed Allow: /hire/*?*isid= for *
  • Removed Allow: /personeel/*?*isid= for *
  • Removed Allow: /reclutamiento/*?*isid= for *
  • Removed Allow: /recruiting/*?*isid= for *
  • … 45 more change(s)
view diff → snapshot
SoundCloud soundcloud.com modified +29 −60
  • Blocked user-agent ‘Brightbot’ entirely
  • 🤖 AI Blocked omgili (Webz.io) entirely
  • Blocked user-agent ‘omgilibot’ entirely
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • … 45 more change(s)
view diff → snapshot
Corriere della Sera corriere.it modified +10 −0
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • Blocked user-agent ‘Diffbot-User’ entirely
  • Blocked user-agent ‘Scrapy’ entirely
  • 🤖 AI Blocked Timpibot (Timpi) entirely
  • 🤖 AI Blocked YouBot (You.com) entirely
view diff → snapshot
SoundCloud soundcloud.com modified +60 −29
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Blocked cohere-ai (Cohere) entirely
  • … 45 more change(s)
view diff → snapshot
BuzzFeed buzzfeed.com modified +1 −0
  • 🤖 AI Added Allow: /rss-feeds/feeds/bf-amazon-ai-grounding.extended.xml for Amazonbot
view diff → snapshot
Figma Community figma.com modified +94 −0
  • 🤖 AI Added Allow: /de-de/resource-library/ki-coding-tools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/resource-library/was-ist-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/resource-library/herramientas-programacion-ia/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/resource-library/que-es-vibe-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/2025-shifting-roles-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/4-ways-for-design-teams-to-chart-new-territory-with-figma-make/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
El País elpais.com modified +14 −0
  • 🤖 AI Blocked Claude-SearchBot (Anthropic) entirely
  • Blocked user-agent ‘Comet’ entirely
  • 🤖 AI Blocked Perplexity-User (Perplexity) entirely
  • 🤖 AI Blocked PerplexityBot (Perplexity) entirely
view diff → snapshot
Figma Community figma.com modified +104 −56
  • 🤖 AI Added Allow: /blog/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/agent-custom-tools-context-skills/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/code-on-the-figma-canvas/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/config-2026-recap/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/future-states/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Ireland (Gov.ie) gov.ie modified +53 −2
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • Blocked user-agent ‘Applebot’ entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Blocked user-agent ‘Baiduspider’ entirely
  • Blocked user-agent ‘Baiduspider-image’ entirely
  • Blocked user-agent ‘Baiduspider-video’ entirely
  • … 13 more change(s)
view diff → snapshot
W.W. Grainger grainger.com modified +188 −167
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • Blocked user-agent ‘AhrefsBot’ entirely
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • Blocked user-agent ‘Baiduspider’ entirely
  • Blocked user-agent ‘Baiduspider-render’ entirely
  • … 19 more change(s)
view diff →
Regeneron Pharmaceuticals regeneron.com modified +92 −1529
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • … 12 more change(s)
view diff →
Evonik evonik.com modified +47 −22343
  • Response is a valid robots.txt again (was an HTML page)
  • Added rules for user-agent ‘*’
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • Added rules for user-agent ‘Bingbot’
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • … 25 more change(s)
view diff →