NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
227this week
All changes 🤖 AI-crawler changes Showing initial archives
Cemex cemex.com modified +46 −2
  • Added Allow: /documents/ for *
  • Added Allow: /favicon.ico for *
  • Added Allow: /image/* for *
  • Added Allow: /o/web/image/* for *
  • Added Allow: /sitemap.xml for *
  • Added Disallow: */web for *
  • … 32 more change(s)
view diff → snapshot
xAI x.ai modified +2 −13
  • 🤖 AI Removed rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Removed rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Removed rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Removed rules for AI crawler Google-Extended (Google)
  • 🤖 AI Removed rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Removed rules for AI crawler PerplexityBot (Perplexity)
view diff → snapshot
Threads threads.net modified +1 −0
  • Added sitemap https://www.threads.com/sitemap/threads_high_potential_posts_v2_sitemap.xml.gz
view diff → snapshot
Figma Community figma.com modified +25 −0
  • 🤖 AI Added Allow: /careers/early-career/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/claude-code-for-designers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/claude-skills-for-product-designers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/enterprise-design-systems/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/figma-mcp-use-cases/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/llm-context-design/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Threads threads.net modified +0 −1
  • Removed sitemap https://www.threads.com/sitemap/threads_high_potential_posts_sitemap.xml.gz
view diff → snapshot
El Corte Ingles elcorteingles.es modified +23 −20
  • Added rules for user-agent ‘AdsBot-Google-Mobile’
  • 🤖 AI Unblocked Claude-User (Anthropic) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Added Disallow: /*?*&analytics_data= for *
  • Added Disallow: /*?*&pdp_ctype= for *
  • Added Disallow: /*?*&term= for *
  • … 45 more change(s)
view diff → snapshot
Yandex yandex.ru modified +5 −2
  • Added Allow: /finance/banks/*/reviews for *
  • Added Allow: /finance/reviews for *
  • Added Disallow: /finance for *
  • Added Disallow: /finance/banks/*/reviews*?* for *
  • Added Disallow: /finance/reviews*?* for *
  • Removed Disallow: /finance*? for *
  • … 1 more change(s)
view diff → snapshot
Glencore glencore.com modified +3 −3
  • Added Disallow: /.rest/api/v1/documents/static/610492b3-3d6e-4622-aff1-4fc0f747b271/Business%2Bconfirmation%2Bgeneral%2Bterms-purchase.pdf for *
  • Added Disallow: /.rest/api/v1/documents/static/72f40bbd-1240-4e41-8177-5fba47c6323f/Business%2Bconfirmation%2Bgeneral%2Bterms-purchase-ext.pdf for *
  • Added Disallow: /.rest/api/v1/documents/static/975c896c-6824-401f-9def-50bc0d31d097/Business%2Bconfirmation%2Bgeneral%2Bterms-sale.pdf for *
  • Removed Disallow: /.rest/api/v1/documents/static/610492b3-3d6e-4622-aff1-4fc0f747b271/Business+confirmation+general+terms-purchase.pdf for *
  • Removed Disallow: /.rest/api/v1/documents/static/72f40bbd-1240-4e41-8177-5fba47c6323f/Business+confirmation+general+terms-purchase-ext.pdf for *
  • Removed Disallow: /.rest/api/v1/documents/static/975c896c-6824-401f-9def-50bc0d31d097/Business+confirmation+general+terms-sale.pdf for *
view diff → snapshot
Bar-Ilan University biu.ac.il removed +0 −207
  • robots.txt is no longer served (HTTP 404)