NeuralCrawl · September 15, 2026
NeuralCrawl
Monitor how companies and governments treat AI crawlers and search engine bots.
Analyze 1273 major US & European companies, governments, social networks and publishers (updated 24/7).
165 of 1201 analysed websites explicitly block at least one AI crawler by name in their robots.txt
9.6%
block
GPTBot (OpenAI) — 115 companies
3820
robots.txt snapshots in the archive
251
policy changes detected in the last 7 days
1204
robots.txt files under 24/7 monitoring
AI bots rejection over time weekly · US 500 cohort
Block ≥1 AI crawler
Block GPTBot
2026-09-04
2026-09-15
12 weekly snapshots with reliable coverage
“13.7% of monitored organisations now explicitly block at least one AI crawler in their robots.txt
OpenAI's GPTBot is named by 115 of them.”
Most blocked AI crawlers share of 1201 analysed websites
Which sectors block AI the most all monitored sites · share blocking ≥1 AI crawler · min 5 companies
Most AI-restrictive companies AI crawlers blocked by name
| # | Company | Sector | Bots blocked |
|---|---|---|---|
| 4 | USA Today | News & Media | 28 |
| 60 | Toronto Star | News & Media | 28 |
| 15 | Politico | News & Media | 27 |
| 20 | DBLP | Other | 27 |
| 38 | Msn | Other | 27 |
| 44 | Daily Mail | News & Media | 27 |
| 104 | IGN | Other | 27 |
| 59 | The Globe and Mail | News & Media | 26 |
| 9 | CNN | News & Media | 25 |
| 42 | The Telegraph | News & Media | 25 |
Change activity robots.txt modifications per week
202
26
201
27
198
28
238
29
234
30
221
31
219
32
215
33
243
34
201
35
224
36
54
37
ISO week numbers. Baseline (first-archive) events excluded.
Latest AI-policy moves from the change feed
- Figma Community — Added Allow: /es-la/blog/source-material-thom-bettridge/$ for CHATGPT-User
- Figma Community — Added Allow: /es-la/blog/source-material-thom-bettridge/$ for Claude-SearchBot
- Figma Community — Added Allow: /es-la/blog/source-material-thom-bettridge/$ for Claude-User
- Figma Community — Added Allow: /es-la/blog/source-material-thom-bettridge/$ for OAI-SearchBot
- Figma Community — Added Allow: /es-la/blog/source-material-thom-bettridge/$ for PerplexityBot
- Reuters — Removed Disallow: /jp/ for ChatGPT-User
- Reuters — Removed Disallow: /jp/ for meta-webindexer
- Lululemon Athletica — Blocked Meta-WebIndexer (Meta) entirely
- xAI — Removed rules for AI crawler Applebot-Extended (Apple)
- xAI — Removed rules for AI crawler ChatGPT-User (OpenAI)
- xAI — Removed rules for AI crawler ClaudeBot (Anthropic)
- xAI — Removed rules for AI crawler Google-Extended (Google)
- xAI — Removed rules for AI crawler GPTBot (OpenAI)
- xAI — Removed rules for AI crawler PerplexityBot (Perplexity)
- Figma Community — Added Allow: /careers/early-career/$ for CHATGPT-User
- Figma Community — Added Allow: /resource-library/claude-code-for-designers/$ for CHATGPT-User
- Figma Community — Added Allow: /resource-library/claude-skills-for-product-designers/$ for CHATGPT-User
- Figma Community — Added Allow: /resource-library/enterprise-design-systems/$ for CHATGPT-User
- Figma Community — Added Allow: /resource-library/figma-mcp-use-cases/$ for CHATGPT-User
- Figma Community — Added Allow: /resource-library/llm-context-design/$ for CHATGPT-User
- Figma Community — Added Allow: /resource-library/ui-ux-design-cost/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/ai-bulk-color-generator/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/ai-component-variant-generation/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/ai-crit-assistant/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/ai-design-qa-assistant/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/ai-design-system-audit/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/ai-edge-case-generator/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/ai-error-state-generator/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/ai-feature-section-generator/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/ai-layout-automation-tool/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/ai-loading-state-generator/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/ai-success-state-generator/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/ai-ui-refiner/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/ai-usability-issue-detector/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/ai-ux-gap-finder/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/apply-auto-layout-at-scale/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/design-system-audits/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/design-to-components/$ for CHATGPT-User
- Figma Community — Added Allow: /solutions/generate-ui-variations/$ for CHATGPT-User
- Figma Community — Added Allow: /careers/early-career/$ for Claude-SearchBot
- Figma Community — Added Allow: /resource-library/claude-code-for-designers/$ for Claude-SearchBot
- Figma Community — Added Allow: /resource-library/claude-skills-for-product-designers/$ for Claude-SearchBot
- Figma Community — Added Allow: /resource-library/enterprise-design-systems/$ for Claude-SearchBot
- Figma Community — Added Allow: /resource-library/figma-mcp-use-cases/$ for Claude-SearchBot
- Figma Community — Added Allow: /resource-library/llm-context-design/$ for Claude-SearchBot
- Figma Community — Added Allow: /resource-library/ui-ux-design-cost/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/ai-bulk-color-generator/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/ai-component-variant-generation/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/ai-crit-assistant/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/ai-design-qa-assistant/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/ai-design-system-audit/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/ai-edge-case-generator/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/ai-error-state-generator/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/ai-feature-section-generator/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/ai-layout-automation-tool/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/ai-loading-state-generator/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/ai-success-state-generator/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/ai-ui-refiner/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/ai-usability-issue-detector/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/ai-ux-gap-finder/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/apply-auto-layout-at-scale/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/design-system-audits/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/design-to-components/$ for Claude-SearchBot
- Figma Community — Added Allow: /solutions/generate-ui-variations/$ for Claude-SearchBot
- El Corte Ingles — Unblocked Claude-User (Anthropic) - rules removed
- El Corte Ingles — Unblocked ClaudeBot (Anthropic) - rules removed
- GitHub — Added rules for AI crawler anthropic-ai (Anthropic)
- GitHub — Blocked Bytespider (ByteDance) entirely
- GitHub — Added rules for AI crawler ClaudeBot (Anthropic)
- GitHub — Added rules for AI crawler GPTBot (OpenAI)
- GitHub — Added rules for AI crawler OAI-SearchBot (OpenAI)
- GitHub — Added rules for AI crawler PerplexityBot (Perplexity)
- Zscaler — Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for Applebot-Extended
- Zscaler — Removed Disallow: /indexNowKey.txt for Applebot-Extended
- Zscaler — Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for ChatGPT-User
- Zscaler — Removed Disallow: /indexNowKey.txt for ChatGPT-User
- Zscaler — Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for Claude-SearchBot
- Zscaler — Removed Disallow: /indexNowKey.txt for Claude-SearchBot
- Zscaler — Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for Claude-User
- Zscaler — Removed Disallow: /indexNowKey.txt for Claude-User
- Zscaler — Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for ClaudeBot
- Zscaler — Removed Disallow: /indexNowKey.txt for ClaudeBot
- Zscaler — Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for Cohere-ai
- Zscaler — Removed Disallow: /indexNowKey.txt for Cohere-ai
- Zscaler — Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for Google-Extended
- Zscaler — Removed Disallow: /indexNowKey.txt for Google-Extended
- Zscaler — Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for GPTBot
- Zscaler — Removed Disallow: /indexNowKey.txt for GPTBot
- Zscaler — Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for OAI-SearchBot
- Zscaler — Removed Disallow: /indexNowKey.txt for OAI-SearchBot
- Zscaler — Added Disallow: /8199ac63-be6c-4cba-9bd9-45500a02d64f.txt for PerplexityBot
- Zscaler — Removed Disallow: /indexNowKey.txt for PerplexityBot
Methodology. robots.txt files are fetched every 6 hours from the
primary domains of every monitored cohort (US & European companies,
governments and social networks). "Blocks" counts organisations that name an
AI crawler in their own User-agent group with
Disallow: /. 1201 of 1273 sites are reachable
and analysed. Browse cohorts on the datasets page and
trends charts. Full crawl health on the
status page.