NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Showing initial archives
EPFL epfl.ch modified +0 −61
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • … 4 more change(s)
view diff → snapshot
AI21 Labs ai21.com modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Iceland (Government.is) government.is modified +0 −61
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +7 −6
  • 🤖 AI Added Allow: /de-de/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /es-la/blog/source-material-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-thom-bettridge/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /es-la/blog/source-material-thom-bettridge/$ for Claude-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-thom-bettridge/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /es-la/blog/source-material-thom-bettridge/$ for PerplexityBot
view diff → snapshot
Reuters reuters.com modified +0 −1
  • Removed Disallow: /jp/ for AASA-Bot
  • Removed Disallow: /jp/ for ADmantX
  • Removed Disallow: /jp/ for AdmiralBot
  • Removed Disallow: /jp/ for AdsBot-Google
  • Removed Disallow: /jp/ for AdsBot-Google-Mobile
  • Removed Disallow: /jp/ for AmazonAdBot
  • … 45 more change(s)
view diff → snapshot
Lululemon Athletica lululemon.com modified +9 −6
  • 🤖 AI Blocked Meta-WebIndexer (Meta) entirely
  • Added Allow: /*/c/*/*?*icid=* for *
  • Added Allow: /c/*/*?*icid=* for *
  • Added Disallow: /*/c/*/*z*z*z* for *
  • Added Disallow: /*/search?Ntt= for *
  • Added Disallow: /c/*/*z*z*z* for *
  • … 4 more change(s)
view diff → snapshot
xAI x.ai modified +2 −13
  • 🤖 AI Removed rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Removed rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Removed rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Removed rules for AI crawler Google-Extended (Google)
  • 🤖 AI Removed rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Removed rules for AI crawler PerplexityBot (Perplexity)
view diff → snapshot
Figma Community figma.com modified +25 −0
  • 🤖 AI Added Allow: /careers/early-career/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/claude-code-for-designers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/claude-skills-for-product-designers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/enterprise-design-systems/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/figma-mcp-use-cases/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/llm-context-design/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
El Corte Ingles elcorteingles.es modified +23 −20
  • Added rules for user-agent ‘AdsBot-Google-Mobile’
  • 🤖 AI Unblocked Claude-User (Anthropic) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Added Disallow: /*?*&analytics_data= for *
  • Added Disallow: /*?*&pdp_ctype= for *
  • Added Disallow: /*?*&term= for *
  • … 45 more change(s)
view diff → snapshot
GitHub github.com modified +140 −9
  • Added rules for user-agent ‘AdsBot-Google’
  • Added rules for user-agent ‘AdsBot-Google-Mobile’
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • … 2 more change(s)
view diff → snapshot
Zscaler zscaler.com modified +1 −1
  • 🤖 AI Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for Applebot-Extended
  • 🤖 AI Removed Disallow: /indexNowKey.txt for Applebot-Extended
  • 🤖 AI Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for ChatGPT-User
  • 🤖 AI Removed Disallow: /indexNowKey.txt for ChatGPT-User
  • 🤖 AI Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for Claude-SearchBot
  • 🤖 AI Removed Disallow: /indexNowKey.txt for Claude-SearchBot
  • … 14 more change(s)
view diff → snapshot
Figma Community figma.com modified +5 −2
  • 🤖 AI Added Allow: /blog/japan-local-data-hosting/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-marisa-chentakul/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /id-id/resource-library/$ for CHATGPT-User
  • 🤖 AI Added Allow: /th-th/resource-library/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /blog/source-material-creative-technologist-marisa-chentakul-meshtimes/$ for CHATGPT-User
  • … 29 more change(s)
view diff → snapshot
Canva canva.com modified +3 −0
  • 🤖 AI Added Allow: /*/ai-connector/ for anthropic-ai
  • 🤖 AI Added Allow: /ai-connector/ for anthropic-ai
  • 🤖 AI Added Allow: /developers/ for anthropic-ai
  • 🤖 AI Added Allow: /*/ai-connector/ for Applebot-Extended
  • 🤖 AI Added Allow: /ai-connector/ for Applebot-Extended
  • 🤖 AI Added Allow: /developers/ for Applebot-Extended
  • … 12 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /webinars/code-connect-to-the-test-coinbase/$ for CHATGPT-User
  • 🤖 AI Added Allow: /webinars/code-connect-to-the-test-coinbase/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /webinars/code-connect-to-the-test-coinbase/$ for Claude-User
  • 🤖 AI Added Allow: /webinars/code-connect-to-the-test-coinbase/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /webinars/code-connect-to-the-test-coinbase/$ for PerplexityBot
view diff → snapshot
Coursera coursera.org modified +260 −2
  • Added Allow: /courses?*&page=1$ for *
  • Added Allow: /courses?*&page=1& for *
  • Added Allow: /courses?*&page=2$ for *
  • Added Allow: /courses?*&page=2& for *
  • Added Allow: /courses?*&page=3$ for *
  • Added Allow: /courses?*&page=3& for *
  • … 45 more change(s)
view diff → snapshot
Corriere della Sera corriere.it modified +15 −0
  • Added rules for user-agent ‘Amzn-SearchBot’
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
  • Added Allow: / for *
  • … 1 more change(s)
view diff → snapshot
Figma Community figma.com modified +6 −0
  • 🤖 AI Added Allow: /de-de/blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/blog/source-material-i-d-magazine-editor-in-chief-thom-bettridge/$ for CHATGPT-User
  • … 24 more change(s)
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/how-coinbase-used-code-connect-to-shrink-token-costs/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/how-we-built-generative-plugins-and-shaders/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-coinbase-used-code-connect-to-shrink-token-costs/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-we-built-generative-plugins-and-shaders/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/how-coinbase-used-code-connect-to-shrink-token-costs/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/how-we-built-generative-plugins-and-shaders/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
DBLP dblp.org modified +71 −1
  • Blocked user-agent ‘AgentTimes’ entirely
  • Blocked user-agent ‘AI2Bot-DeepResearchEval’ entirely
  • Blocked user-agent ‘AIWebIndex’ entirely
  • Blocked user-agent ‘amazon-kendra’ entirely
  • Blocked user-agent ‘amazon-QBusiness’ entirely
  • Blocked user-agent ‘AmazonBuyForMe’ entirely
  • … 45 more change(s)
view diff → snapshot
SAP sap.com modified +45 −8
  • 🤖 AI Added Disallow: /fetch for Amazonbot
  • 🤖 AI Added Disallow: /proxy for Amazonbot
  • 🤖 AI Added Disallow: /sharedlibs/ for Amazonbot
  • 🤖 AI Added Disallow: /fetch for ChatGPT-User
  • 🤖 AI Added Disallow: /proxy for ChatGPT-User
  • 🤖 AI Added Disallow: /sharedlibs/ for ChatGPT-User
  • … 12 more change(s)
view diff → snapshot
Commonwealth Bank commbank.com.au modified +0 −2
  • Removed Disallow: /personal/eform/ for *
  • 🤖 AI Removed Disallow: /personal/eform/ for Amazonbot
  • 🤖 AI Removed Disallow: /personal/eform/ for Applebot-Extended
  • 🤖 AI Removed Disallow: /personal/eform/ for Bytespider
  • 🤖 AI Removed Disallow: /personal/eform/ for CCBot
  • 🤖 AI Removed Disallow: /personal/eform/ for ChatGPT-User
  • … 12 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/source-material-creative-technologist-marisa-chentakul-meshtimes/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-creative-technologist-marisa-chentakul-meshtimes/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-creative-technologist-marisa-chentakul-meshtimes/$ for Claude-User
  • 🤖 AI Added Allow: /blog/source-material-creative-technologist-marisa-chentakul-meshtimes/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-creative-technologist-marisa-chentakul-meshtimes/$ for PerplexityBot
view diff → snapshot
Moz moz.com modified +51 −4
  • Added rules for user-agent ‘AhrefsBot’
  • Added rules for user-agent ‘DDM-DCipher’
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • Added rules for user-agent ‘rogerbot’
  • Unblocked user-agent ‘*’
  • Added Allow: /learn/ for *
  • … 31 more change(s)
view diff → snapshot
Tripadvisor tripadvisor.com modified +0 −2
  • Removed Disallow: /OwnerEmailManage for *
  • Removed Disallow: /developerslegacy for *
  • Removed Disallow: /OwnerEmailManage for applebot
  • Removed Disallow: /developerslegacy for applebot
  • Removed Disallow: /OwnerEmailManage for Baiduspider
  • Removed Disallow: /developerslegacy for Baiduspider
  • … 14 more change(s)
view diff → snapshot
Figma Community figma.com modified +6 −0
  • 🤖 AI Added Allow: /id-id/resource-library/alat-desain-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/resource-library/strumenti-design-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/resource-library/ai-design-tools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /nl-nl/resource-library/ai-ontwerptools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pl-pl/resource-library/narzedzia-projektowania-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /th-th/resource-library/ai-design-tools/$ for CHATGPT-User
  • … 24 more change(s)
view diff → snapshot
Moz moz.com modified +4 −54
  • Removed all rules for user-agent ‘AhrefsBot’
  • Removed all rules for user-agent ‘DDM-DCipher’
  • 🤖 AI Removed rules for AI crawler GPTBot (OpenAI)
  • Removed all rules for user-agent ‘rogerbot’
  • Blocked user-agent ‘*’ entirely
  • Removed Allow: /learn/ for *
  • … 31 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /pl-pl/rozwiazania/generator-systemow-projektowania-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pl-pl/rozwiazania/generator-systemow-projektowania-ai/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /pl-pl/rozwiazania/generator-systemow-projektowania-ai/$ for Claude-User
  • 🤖 AI Added Allow: /pl-pl/rozwiazania/generator-systemow-projektowania-ai/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /pl-pl/rozwiazania/generator-systemow-projektowania-ai/$ for PerplexityBot
view diff → snapshot
Wired wired.com modified +35 −26
  • Blocked user-agent ‘Bravebot’ entirely
  • Blocked user-agent ‘Brightbot 1.0’ entirely
  • Blocked user-agent ‘factset_spyderbot’ entirely
  • Blocked user-agent ‘FirecrawlAgent’ entirely
  • Blocked user-agent ‘Scrapy’ entirely
  • Blocked user-agent ‘Shap-User’ entirely
  • … 3 more change(s)
view diff → snapshot
The New Yorker newyorker.com modified +35 −26
  • Blocked user-agent ‘Bravebot’ entirely
  • Blocked user-agent ‘Brightbot 1.0’ entirely
  • Blocked user-agent ‘factset_spyderbot’ entirely
  • Blocked user-agent ‘FirecrawlAgent’ entirely
  • Blocked user-agent ‘Scrapy’ entirely
  • Blocked user-agent ‘Shap-User’ entirely
  • … 3 more change(s)
view diff → snapshot