NeuralCrawl · August 01, 2026

NeuralCrawl

Monitor how companies and governments treat AI crawlers and search engine bots.

Analyze 1273 major US & European companies, governments, social networks and publishers (updated 24/7).

159 of 1195 analysed websites explicitly block at least one AI crawler by name in their robots.txt

9.4% block GPTBot (OpenAI) — 112 companies

2434 robots.txt snapshots in the archive

246 policy changes detected in the last 7 days

1198 robots.txt files under 24/7 monitoring

AI bots rejection over time weekly · US 500 cohort

Block ≥1 AI crawler Block GPTBot

2026-07-21 2026-08-01

12 weekly snapshots with reliable coverage

“13.3% of monitored organisations now explicitly block at least one AI crawler in their robots.txt
OpenAI's GPTBot is named by 112 of them.”
— Elie Berreby, August 01, 2026 · cohort us-500 · trends

Most blocked AI crawlers share of 1195 analysed websites

GPTBot OpenAI 112 (9.4%) CCBot Common Crawl 108 (9.0%) ClaudeBot Anthropic 103 (8.6%) Bytespider ByteDance 98 (8.2%) Google-Extended Google 95 (7.9%) meta-externalagent Meta 89 (7.4%) Applebot-Extended Apple 84 (7.0%) Amazonbot Amazon 74 (6.2%) anthropic-ai Anthropic 69 (5.8%) Diffbot Diffbot 65 (5.4%)

Which sectors block AI the most all monitored sites · share blocking ≥1 AI crawler · min 5 companies

News & Media 55.1% (43/78)

Social & Communities 50.0% (6/12)

Entertainment & Sports 24.2% (8/33)

Telecom 13.0% (3/23)

Retail & E-commerce 11.9% (12/101)

Technology & Software 10.6% (25/236)

Business & Professional Services 9.1% (1/11)

Education & Research 9.0% (6/67)

Energy & Utilities 7.0% (5/71)

Travel & Hospitality 5.0% (1/20)

Finance & Insurance 3.6% (5/139)

Industrials & Manufacturing 3.3% (4/120)

Most AI-restrictive companies AI crawlers blocked by name

#	Company	Sector	Bots blocked
4	USA Today	News & Media	28
60	Toronto Star	News & Media	28
15	Politico	News & Media	27
38	Msn	Other	27
44	Daily Mail	News & Media	27
104	IGN	Other	27
42	The Telegraph	News & Media	26
59	The Globe and Mail	News & Media	26
9	CNN	News & Media	25
56	Nikkei	Finance & Insurance	25

Change activity robots.txt modifications per week

42 24

173 25

202 26

201 27

198 28

238 29

206 30

ISO week numbers. Baseline (first-archive) events excluded.

Latest AI-policy moves from the change feed

Figma Community — Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for CHATGPT-User 16h ago
Figma Community — Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for Claude-SearchBot 16h ago
Figma Community — Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for OAI-SearchBot 16h ago
Figma Community — Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for PerplexityBot 16h ago
Morgan Stanley — Added rules for AI crawler Amazonbot (Amazon) 22h ago
Morgan Stanley — Added rules for AI crawler anthropic-ai (Anthropic) 22h ago
Morgan Stanley — Added rules for AI crawler Applebot-Extended (Apple) 22h ago
Morgan Stanley — Added rules for AI crawler ChatGPT-User (OpenAI) 22h ago
Morgan Stanley — Added rules for AI crawler Claude-Web (Anthropic) 22h ago
Morgan Stanley — Added rules for AI crawler ClaudeBot (Anthropic) 22h ago
Morgan Stanley — Added rules for AI crawler cohere-ai (Cohere) 22h ago
Morgan Stanley — Added rules for AI crawler FacebookBot (Meta) 22h ago
Morgan Stanley — Added rules for AI crawler Google-Extended (Google) 22h ago
Morgan Stanley — Added rules for AI crawler GPTBot (OpenAI) 22h ago
Morgan Stanley — Added rules for AI crawler meta-externalagent (Meta) 22h ago
Morgan Stanley — Added rules for AI crawler OAI-SearchBot (OpenAI) 22h ago
Morgan Stanley — Added rules for AI crawler PerplexityBot (Perplexity) 22h ago
Morgan Stanley — Added rules for AI crawler YouBot (You.com) 22h ago
Figma Community — Added Allow: /de-de/blog/how-decagon-uses-ai-for-design-system-saturation/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /de-de/blog/sightlines-issue-no1/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /de-de/blog/steal-this-template-bring-a-user-persona-to-life-with-figma-weave/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /de-de/blog/workflow-lab-deploying-designs-directly-with-figma-make/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /es-es/blog/how-decagon-uses-ai-for-design-system-saturation/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /es-es/blog/sightlines-issue-no1/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /es-es/blog/steal-this-template-bring-a-user-persona-to-life-with-figma-weave/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /es-la/blog/how-decagon-uses-ai-for-design-system-saturation/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /es-la/blog/sightlines-issue-no1/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /fr-fr/blog/4-ways-were-using-our-mcp-server-at-figma/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /fr-fr/blog/how-decagon-uses-ai-for-design-system-saturation/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /fr-fr/blog/sightlines-issue-no1/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /fr-fr/blog/steal-this-template-bring-a-user-persona-to-life-with-figma-weave/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /ja-jp/blog/how-decagon-uses-ai-for-design-system-saturation/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /ja-jp/blog/sightlines-issue-no1/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /ja-jp/blog/steal-this-template-bring-a-user-persona-to-life-with-figma-weave/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /ko-kr/blog/how-decagon-uses-ai-for-design-system-saturation/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /ko-kr/blog/sightlines-issue-no1/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /ko-kr/blog/steal-this-template-bring-a-user-persona-to-life-with-figma-weave/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /pt-br/blog/how-decagon-uses-ai-for-design-system-saturation/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /pt-br/blog/sightlines-issue-no1/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /pt-br/blog/steal-this-template-bring-a-user-persona-to-life-with-figma-weave/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /webinars/workflow-lab-adyen-mcp/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /webinars/workflow-lab-context-to-code/$ for CHATGPT-User 22h ago
Figma Community — Added Allow: /de-de/blog/how-decagon-uses-ai-for-design-system-saturation/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /de-de/blog/sightlines-issue-no1/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /de-de/blog/steal-this-template-bring-a-user-persona-to-life-with-figma-weave/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /de-de/blog/workflow-lab-deploying-designs-directly-with-figma-make/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /es-es/blog/how-decagon-uses-ai-for-design-system-saturation/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /es-es/blog/sightlines-issue-no1/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /es-es/blog/steal-this-template-bring-a-user-persona-to-life-with-figma-weave/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /es-la/blog/how-decagon-uses-ai-for-design-system-saturation/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /es-la/blog/sightlines-issue-no1/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /fr-fr/blog/4-ways-were-using-our-mcp-server-at-figma/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /fr-fr/blog/how-decagon-uses-ai-for-design-system-saturation/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /fr-fr/blog/sightlines-issue-no1/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /fr-fr/blog/steal-this-template-bring-a-user-persona-to-life-with-figma-weave/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /ja-jp/blog/how-decagon-uses-ai-for-design-system-saturation/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /ja-jp/blog/sightlines-issue-no1/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /ja-jp/blog/steal-this-template-bring-a-user-persona-to-life-with-figma-weave/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /ko-kr/blog/how-decagon-uses-ai-for-design-system-saturation/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /ko-kr/blog/sightlines-issue-no1/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /ko-kr/blog/steal-this-template-bring-a-user-persona-to-life-with-figma-weave/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /pt-br/blog/how-decagon-uses-ai-for-design-system-saturation/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /pt-br/blog/sightlines-issue-no1/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /pt-br/blog/steal-this-template-bring-a-user-persona-to-life-with-figma-weave/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /webinars/workflow-lab-adyen-mcp/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /webinars/workflow-lab-context-to-code/$ for Claude-SearchBot 22h ago
Figma Community — Added Allow: /de-de/blog/how-decagon-uses-ai-for-design-system-saturation/$ for OAI-SearchBot 22h ago
Figma Community — Added Allow: /de-de/blog/sightlines-issue-no1/$ for OAI-SearchBot 22h ago
Kohl's — Blocked Meta-WebIndexer (Meta) entirely 1d ago
Figma Community — Added Allow: /blog/properties-panel-and-annotations-now-in-figma-make/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-annotation-generator/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-background-generator/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-card-layout-generator/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-color-palette-generator/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-empty-state-generator/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-file-cleaner/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-form-layout-generator/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-gradient-generator/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-grid-generator/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-icon-generator/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-illustration-style-changer/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-modal-generator/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-shadow-generator/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-sidebar-generator/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-table-layout-generator/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-typography-pairer/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /blog/properties-panel-and-annotations-now-in-figma-make/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-annotation-generator/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-background-generator/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-card-layout-generator/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-color-palette-generator/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-empty-state-generator/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-file-cleaner/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-form-layout-generator/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-gradient-generator/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-grid-generator/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-icon-generator/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-illustration-style-changer/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-modal-generator/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-shadow-generator/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-sidebar-generator/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-table-layout-generator/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-typography-pairer/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /blog/properties-panel-and-annotations-now-in-figma-make/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-annotation-generator/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-background-generator/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-card-layout-generator/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-color-palette-generator/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-empty-state-generator/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-file-cleaner/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-form-layout-generator/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-gradient-generator/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-grid-generator/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-icon-generator/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-illustration-style-changer/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-modal-generator/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-shadow-generator/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-sidebar-generator/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-table-layout-generator/$ for OAI-SearchBot 1d ago
Claude — Blocked Google-Extended (Google) entirely 1d ago
Figma Community — Added Allow: /solutions/ai-frame-namer/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-page-organizer/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-variable-namer/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /solutions/ai-frame-namer/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-page-organizer/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-variable-namer/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-frame-namer/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-page-organizer/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-variable-namer/$ for OAI-SearchBot 1d ago
Figma Community — Added Allow: /solutions/ai-frame-namer/$ for PerplexityBot 1d ago
Figma Community — Added Allow: /solutions/ai-page-organizer/$ for PerplexityBot 1d ago
Figma Community — Added Allow: /solutions/ai-variable-namer/$ for PerplexityBot 1d ago
Figma Community — Added Allow: /blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User 2d ago
Figma Community — Added Allow: /de-de/customers/duotone-saves-hours/$ for CHATGPT-User 2d ago
Figma Community — Added Allow: /es-es/customers/duotone-saves-hours/$ for CHATGPT-User 2d ago
Figma Community — Added Allow: /es-la/customers/duotone-saves-hours/$ for CHATGPT-User 2d ago
Figma Community — Added Allow: /fr-fr/customers/duotone-saves-hours/$ for CHATGPT-User 2d ago
Figma Community — Added Allow: /it-it/customers/duotone-saves-hours/$ for CHATGPT-User 2d ago
Figma Community — Added Allow: /ja-jp/customers/duotone-saves-hours/$ for CHATGPT-User 2d ago
Figma Community — Added Allow: /blog/how-we-secure-figmas-internal-systems-with-agents/$ for Claude-SearchBot 2d ago
Figma Community — Added Allow: /de-de/customers/duotone-saves-hours/$ for Claude-SearchBot 2d ago
Figma Community — Added Allow: /es-es/customers/duotone-saves-hours/$ for Claude-SearchBot 2d ago
Figma Community — Added Allow: /es-la/customers/duotone-saves-hours/$ for Claude-SearchBot 2d ago
Figma Community — Added Allow: /fr-fr/customers/duotone-saves-hours/$ for Claude-SearchBot 2d ago
Figma Community — Added Allow: /it-it/customers/duotone-saves-hours/$ for Claude-SearchBot 2d ago
Figma Community — Added Allow: /ja-jp/customers/duotone-saves-hours/$ for Claude-SearchBot 2d ago
Figma Community — Added Allow: /blog/how-we-secure-figmas-internal-systems-with-agents/$ for OAI-SearchBot 2d ago
Figma Community — Added Allow: /de-de/customers/duotone-saves-hours/$ for OAI-SearchBot 2d ago
Figma Community — Added Allow: /es-es/customers/duotone-saves-hours/$ for OAI-SearchBot 2d ago
Figma Community — Added Allow: /es-la/customers/duotone-saves-hours/$ for OAI-SearchBot 2d ago
Figma Community — Added Allow: /fr-fr/customers/duotone-saves-hours/$ for OAI-SearchBot 2d ago
Figma Community — Added Allow: /it-it/customers/duotone-saves-hours/$ for OAI-SearchBot 2d ago
Figma Community — Added Allow: /ja-jp/customers/duotone-saves-hours/$ for OAI-SearchBot 2d ago
Figma Community — Added Allow: /blog/how-we-secure-figmas-internal-systems-with-agents/$ for PerplexityBot 2d ago
Figma Community — Added Allow: /de-de/customers/duotone-saves-hours/$ for PerplexityBot 2d ago
Figma Community — Added Allow: /es-es/customers/duotone-saves-hours/$ for PerplexityBot 2d ago
Figma Community — Added Allow: /es-la/customers/duotone-saves-hours/$ for PerplexityBot 2d ago
Figma Community — Added Allow: /fr-fr/customers/duotone-saves-hours/$ for PerplexityBot 2d ago
Figma Community — Added Allow: /it-it/customers/duotone-saves-hours/$ for PerplexityBot 2d ago
Figma Community — Added Allow: /ja-jp/customers/duotone-saves-hours/$ for PerplexityBot 2d ago

Methodology. robots.txt files are fetched every 6 hours from the primary domains of every monitored cohort (US & European companies, governments and social networks). "Blocks" counts organisations that name an AI crawler in their own User-agent group with Disallow: /. 1195 of 1273 sites are reachable and analysed. Browse cohorts on the datasets page and trends charts. Full crawl health on the status page.