NeuralCrawl · November 07, 2026
NeuralCrawl
Monitor how companies and governments treat AI crawlers and search engine bots.
Analyze 1273 major US & European companies, governments, social networks and publishers (updated 24/7).
166 of 1203 analysed websites explicitly block at least one AI crawler by name in their robots.txt
9.3%
block
GPTBot (OpenAI) — 112 companies
5416
robots.txt snapshots in the archive
237
policy changes detected in the last 7 days
1206
robots.txt files under 24/7 monitoring
AI bots rejection over time weekly · US 500 cohort
Block ≥1 AI crawler
Block GPTBot
2026-10-27
2026-11-07
12 weekly snapshots with reliable coverage
“13.8% of monitored organisations now explicitly block at least one AI crawler in their robots.txt
OpenAI's GPTBot is named by 112 of them.”
Most blocked AI crawlers share of 1203 analysed websites
Which sectors block AI the most all monitored sites · share blocking ≥1 AI crawler · min 5 companies
Most AI-restrictive companies AI crawlers blocked by name
| # | Company | Sector | Bots blocked |
|---|---|---|---|
| 4 | USA Today | News & Media | 28 |
| 44 | Daily Mail | News & Media | 28 |
| 60 | Toronto Star | News & Media | 28 |
| 15 | Politico | News & Media | 27 |
| 38 | Msn | Other | 27 |
| 104 | IGN | Other | 27 |
| 59 | The Globe and Mail | News & Media | 26 |
| 42 | The Telegraph | News & Media | 25 |
| 54 | Corriere della Sera | News & Media | 25 |
| 56 | Nikkei | Finance & Insurance | 25 |
Change activity robots.txt modifications per week
215
33
243
34
201
35
224
36
205
37
225
38
203
39
219
40
214
41
190
42
216
43
197
44
ISO week numbers. Baseline (first-archive) events excluded.
Latest AI-policy moves from the change feed
- GitHub — Added rules for AI crawler ChatGPT-User (OpenAI)
- GitHub — Added rules for AI crawler Claude-SearchBot (Anthropic)
- GitHub — Added rules for AI crawler Claude-User (Anthropic)
- GitHub — Added rules for AI crawler Perplexity-User (Perplexity)
- GitHub — Removed rules for AI crawler anthropic-ai (Anthropic)
- GitHub — Added Allow: /enterprise/contact/data-residency?* for ClaudeBot
- GitHub — Added Allow: /enterprise/contact?* for ClaudeBot
- GitHub — Added Allow: /features/codespaces?* for ClaudeBot
- GitHub — Added Allow: /features/copilot/plans?* for ClaudeBot
- GitHub — Added Allow: /security/advanced-security/demo?* for ClaudeBot
- GitHub — Added Allow: /security/contact-sales?* for ClaudeBot
- GitHub — Added Allow: /security/plans?* for ClaudeBot
- GitHub — Added Allow: /enterprise/contact/data-residency?* for GPTBot
- GitHub — Added Allow: /enterprise/contact?* for GPTBot
- GitHub — Added Allow: /features/codespaces?* for GPTBot
- GitHub — Added Allow: /features/copilot/plans?* for GPTBot
- GitHub — Added Allow: /security/advanced-security/demo?* for GPTBot
- GitHub — Added Allow: /security/contact-sales?* for GPTBot
- GitHub — Added Allow: /security/plans?* for GPTBot
- GitHub — Added Allow: /enterprise/contact/data-residency?* for OAI-SearchBot
- GitHub — Added Allow: /enterprise/contact?* for OAI-SearchBot
- GitHub — Added Allow: /features/codespaces?* for OAI-SearchBot
- GitHub — Added Allow: /features/copilot/plans?* for OAI-SearchBot
- GitHub — Added Allow: /security/advanced-security/demo?* for OAI-SearchBot
- GitHub — Added Allow: /security/contact-sales?* for OAI-SearchBot
- GitHub — Added Allow: /security/plans?* for OAI-SearchBot
- GitHub — Added Allow: /enterprise/contact/data-residency?* for PerplexityBot
- GitHub — Added Allow: /enterprise/contact?* for PerplexityBot
- GitHub — Added Allow: /features/codespaces?* for PerplexityBot
- GitHub — Added Allow: /features/copilot/plans?* for PerplexityBot
- GitHub — Added Allow: /security/advanced-security/demo?* for PerplexityBot
- GitHub — Added Allow: /security/contact-sales?* for PerplexityBot
- GitHub — Added Allow: /security/plans?* for PerplexityBot
- Advance Auto Parts — Blocked Applebot-Extended (Apple) entirely
- Advance Auto Parts — Blocked Bytespider (ByteDance) entirely
- Advance Auto Parts — Blocked CCBot (Common Crawl) entirely
- Advance Auto Parts — Blocked ClaudeBot (Anthropic) entirely
- Advance Auto Parts — Blocked Google-Extended (Google) entirely
- Advance Auto Parts — Blocked GPTBot (OpenAI) entirely
- Advance Auto Parts — Blocked meta-externalagent (Meta) entirely
- Figma Community — Added Allow: /blog/in-conversation-dylan-field-and-david-friedberg/$ for CHATGPT-User
- Figma Community — Added Allow: /de-de/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
- Figma Community — Added Allow: /es-es/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
- Figma Community — Added Allow: /es-la/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
- Figma Community — Added Allow: /fr-fr/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
- Figma Community — Added Allow: /ko-kr/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
- Figma Community — Added Allow: /pl-pl/mcp-catalog/$ for CHATGPT-User
- Figma Community — Added Allow: /pt-br/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
- Figma Community — Added Allow: /blog/in-conversation-dylan-field-and-david-friedberg/$ for Claude-SearchBot
- Figma Community — Added Allow: /de-de/blog/hello-kitty-and-friends-in-figma/$ for Claude-SearchBot
- Figma Community — Added Allow: /es-es/blog/hello-kitty-and-friends-in-figma/$ for Claude-SearchBot
- Figma Community — Added Allow: /es-la/blog/hello-kitty-and-friends-in-figma/$ for Claude-SearchBot
- Figma Community — Added Allow: /fr-fr/blog/hello-kitty-and-friends-in-figma/$ for Claude-SearchBot
- Figma Community — Added Allow: /ko-kr/blog/hello-kitty-and-friends-in-figma/$ for Claude-SearchBot
- Figma Community — Added Allow: /pl-pl/mcp-catalog/$ for Claude-SearchBot
- Figma Community — Added Allow: /pt-br/blog/hello-kitty-and-friends-in-figma/$ for Claude-SearchBot
- Figma Community — Added Allow: /blog/in-conversation-dylan-field-and-david-friedberg/$ for Claude-User
- Figma Community — Added Allow: /de-de/blog/hello-kitty-and-friends-in-figma/$ for Claude-User
- Figma Community — Added Allow: /es-es/blog/hello-kitty-and-friends-in-figma/$ for Claude-User
- Figma Community — Added Allow: /es-la/blog/hello-kitty-and-friends-in-figma/$ for Claude-User
- Figma Community — Added Allow: /fr-fr/blog/hello-kitty-and-friends-in-figma/$ for Claude-User
- Figma Community — Added Allow: /ko-kr/blog/hello-kitty-and-friends-in-figma/$ for Claude-User
- Figma Community — Added Allow: /pl-pl/mcp-catalog/$ for Claude-User
- Figma Community — Added Allow: /pt-br/blog/hello-kitty-and-friends-in-figma/$ for Claude-User
- Figma Community — Added Allow: /blog/in-conversation-dylan-field-and-david-friedberg/$ for OAI-SearchBot
- Figma Community — Added Allow: /de-de/blog/hello-kitty-and-friends-in-figma/$ for OAI-SearchBot
- Figma Community — Added Allow: /es-es/blog/hello-kitty-and-friends-in-figma/$ for OAI-SearchBot
- Figma Community — Added Allow: /es-la/blog/hello-kitty-and-friends-in-figma/$ for OAI-SearchBot
- Figma Community — Added Allow: /fr-fr/blog/hello-kitty-and-friends-in-figma/$ for OAI-SearchBot
- Figma Community — Added Allow: /ko-kr/blog/hello-kitty-and-friends-in-figma/$ for OAI-SearchBot
- Figma Community — Added Allow: /pl-pl/mcp-catalog/$ for OAI-SearchBot
- Figma Community — Added Allow: /pt-br/blog/hello-kitty-and-friends-in-figma/$ for OAI-SearchBot
- Figma Community — Added Allow: /blog/in-conversation-dylan-field-and-david-friedberg/$ for PerplexityBot
- Figma Community — Added Allow: /de-de/blog/hello-kitty-and-friends-in-figma/$ for PerplexityBot
- Figma Community — Added Allow: /es-es/blog/hello-kitty-and-friends-in-figma/$ for PerplexityBot
- Figma Community — Added Allow: /es-la/blog/hello-kitty-and-friends-in-figma/$ for PerplexityBot
- Figma Community — Added Allow: /fr-fr/blog/hello-kitty-and-friends-in-figma/$ for PerplexityBot
- Figma Community — Added Allow: /ko-kr/blog/hello-kitty-and-friends-in-figma/$ for PerplexityBot
- Figma Community — Added Allow: /pl-pl/mcp-catalog/$ for PerplexityBot
- Figma Community — Added Allow: /pt-br/blog/hello-kitty-and-friends-in-figma/$ for PerplexityBot
- Advance Auto Parts — Unblocked Applebot-Extended (Apple) - rules removed
- Advance Auto Parts — Unblocked Bytespider (ByteDance) - rules removed
- Advance Auto Parts — Unblocked CCBot (Common Crawl) - rules removed
- Advance Auto Parts — Unblocked ClaudeBot (Anthropic) - rules removed
- Advance Auto Parts — Unblocked Google-Extended (Google) - rules removed
- Advance Auto Parts — Unblocked GPTBot (OpenAI) - rules removed
- Advance Auto Parts — Unblocked meta-externalagent (Meta) - rules removed
- Figma Community — Added Allow: /de-de/legal/privacy/$ for CHATGPT-User
- Figma Community — Added Allow: /de-de/legal/tos/$ for CHATGPT-User
- Figma Community — Added Allow: /fr-fr/legal/privacy/$ for CHATGPT-User
- Figma Community — Added Allow: /fr-fr/legal/tos/$ for CHATGPT-User
- Figma Community — Added Allow: /hi-in/legal/$ for CHATGPT-User
- Figma Community — Added Allow: /id-id/legal/$ for CHATGPT-User
- Figma Community — Added Allow: /it-it/legal/$ for CHATGPT-User
- Figma Community — Added Allow: /ja-jp/legal/privacy/$ for CHATGPT-User
- Figma Community — Added Allow: /ja-jp/legal/tos/$ for CHATGPT-User
- Figma Community — Added Allow: /pl-pl/legal/$ for CHATGPT-User
- Figma Community — Added Allow: /th-th/legal/$ for CHATGPT-User
- Figma Community — Added Allow: /de-de/legal/privacy/$ for Claude-SearchBot
- Figma Community — Added Allow: /de-de/legal/tos/$ for Claude-SearchBot
- Figma Community — Added Allow: /fr-fr/legal/privacy/$ for Claude-SearchBot
- Figma Community — Added Allow: /fr-fr/legal/tos/$ for Claude-SearchBot
- Figma Community — Added Allow: /hi-in/legal/$ for Claude-SearchBot
- Figma Community — Added Allow: /id-id/legal/$ for Claude-SearchBot
- Figma Community — Added Allow: /it-it/legal/$ for Claude-SearchBot
- Figma Community — Added Allow: /ja-jp/legal/privacy/$ for Claude-SearchBot
- Figma Community — Added Allow: /ja-jp/legal/tos/$ for Claude-SearchBot
- Figma Community — Added Allow: /pl-pl/legal/$ for Claude-SearchBot
- Figma Community — Added Allow: /th-th/legal/$ for Claude-SearchBot
- Figma Community — Added Allow: /de-de/legal/privacy/$ for Claude-User
- Figma Community — Added Allow: /de-de/legal/tos/$ for Claude-User
- Figma Community — Added Allow: /fr-fr/legal/privacy/$ for Claude-User
- Figma Community — Added Allow: /fr-fr/legal/tos/$ for Claude-User
- Figma Community — Added Allow: /hi-in/legal/$ for Claude-User
- Figma Community — Added Allow: /id-id/legal/$ for Claude-User
- Figma Community — Added Allow: /it-it/legal/$ for Claude-User
- Figma Community — Added Allow: /ja-jp/legal/privacy/$ for Claude-User
- Figma Community — Added Allow: /ja-jp/legal/tos/$ for Claude-User
- Figma Community — Added Allow: /pl-pl/legal/$ for Claude-User
- Figma Community — Added Allow: /th-th/legal/$ for Claude-User
- Figma Community — Added Allow: /de-de/legal/privacy/$ for OAI-SearchBot
- Figma Community — Added Allow: /de-de/legal/tos/$ for OAI-SearchBot
- Figma Community — Added Allow: /fr-fr/legal/privacy/$ for OAI-SearchBot
- Figma Community — Added Allow: /fr-fr/legal/tos/$ for OAI-SearchBot
- Figma Community — Added Allow: /hi-in/legal/$ for OAI-SearchBot
- Figma Community — Added Allow: /id-id/legal/$ for OAI-SearchBot
- Figma Community — Added Allow: /it-it/legal/$ for OAI-SearchBot
- Figma Community — Added Allow: /ja-jp/legal/privacy/$ for OAI-SearchBot
- Figma Community — Added Allow: /ja-jp/legal/tos/$ for OAI-SearchBot
- Figma Community — Added Allow: /pl-pl/legal/$ for OAI-SearchBot
- Figma Community — Added Allow: /th-th/legal/$ for OAI-SearchBot
- Figma Community — Added Allow: /de-de/legal/privacy/$ for PerplexityBot
- Figma Community — Added Allow: /de-de/legal/tos/$ for PerplexityBot
- Figma Community — Added Allow: /fr-fr/legal/privacy/$ for PerplexityBot
- Figma Community — Added Allow: /fr-fr/legal/tos/$ for PerplexityBot
- Figma Community — Added Allow: /hi-in/legal/$ for PerplexityBot
- Figma Community — Added Allow: /id-id/legal/$ for PerplexityBot
- Delta Air Lines — Removed rules for AI crawler Amazonbot (Amazon)
- Delta Air Lines — Removed rules for AI crawler anthropic-ai (Anthropic)
- Delta Air Lines — Removed rules for AI crawler Applebot-Extended (Apple)
- Delta Air Lines — Removed rules for AI crawler Bytespider (ByteDance)
- Delta Air Lines — Removed rules for AI crawler CCBot (Common Crawl)
- Delta Air Lines — Removed rules for AI crawler ChatGPT-User (OpenAI)
- Delta Air Lines — Removed rules for AI crawler ClaudeBot (Anthropic)
- Delta Air Lines — Removed rules for AI crawler cohere-ai (Cohere)
- Delta Air Lines — Removed rules for AI crawler Diffbot (Diffbot)
- Delta Air Lines — Removed rules for AI crawler DuckAssistBot (DuckDuckGo)
- Delta Air Lines — Removed rules for AI crawler FacebookBot (Meta)
- Delta Air Lines — Removed rules for AI crawler Google-Extended (Google)
- Delta Air Lines — Removed rules for AI crawler GPTBot (OpenAI)
- Delta Air Lines — Removed rules for AI crawler meta-externalagent (Meta)
- Delta Air Lines — Removed rules for AI crawler meta-externalfetcher (Meta)
- Delta Air Lines — Removed rules for AI crawler OAI-SearchBot (OpenAI)
- Delta Air Lines — Removed rules for AI crawler PerplexityBot (Perplexity)
- Delta Air Lines — Removed rules for AI crawler YouBot (You.com)
- The Telegraph — Added Allow: /bah-avis/usa-sports-road-trip-map for AI2Bot
- The Telegraph — Added Allow: /bespoke-content/ for AI2Bot
- The Telegraph — Added Allow: /betting/ for AI2Bot
- The Telegraph — Added Allow: /brand-usa for AI2Bot
- The Telegraph — Added Allow: /branded-content/ for AI2Bot
- The Telegraph — Added Allow: /british-airways-holidays/cyprus-hotel-map for AI2Bot
- The Telegraph — Added Allow: /business/ai-era-job-skills/ for AI2Bot
- The Telegraph — Added Allow: /business/ai-workflow-transformation/ for AI2Bot
- The Telegraph — Added Allow: /business/future-hr-technology/ for AI2Bot
- The Telegraph — Added Allow: /business/golf-dp-world-tour/ for AI2Bot
- The Telegraph — Added Allow: /business/search-marketing/ for AI2Bot
- The Telegraph — Added Allow: /business/waste-management/ for AI2Bot
- The Telegraph — Added Allow: /deals/ for AI2Bot
- The Telegraph — Added Allow: /environment/woodland-action/ for AI2Bot
- The Telegraph — Added Allow: /fashion/joseph-ribkoff-autumn-collection/ for AI2Bot
- The Telegraph — Added Allow: /financial-services/ for AI2Bot
- The Telegraph — Added Allow: /food-and-drink/m-and-s-seasonal-favourites/ for AI2Bot
- The Telegraph — Added Allow: /food-and-drink/m-and-s-seasonal-favourites/organic-range/ for AI2Bot
- The Telegraph — Added Allow: /food-and-drink/maldon-salt-perfect-pyramid-challenge/ for AI2Bot
- The Telegraph — Added Allow: /food-and-drink/spanish-dining-uk/ for AI2Bot
- The Telegraph — Added Allow: /football/global-football-industry-awards/ for AI2Bot
- The Telegraph — Added Allow: /health-fitness/how-to-choose-private-hospital-hca-healthcare/ for AI2Bot
- The Telegraph — Added Allow: /interiors/next-home-autumn-interiors-trends/ for AI2Bot
- The Telegraph — Added Allow: /junior-golf/ for AI2Bot
- The Telegraph — Added Allow: /lifestyle-solutions/ for AI2Bot
- The Telegraph — Added Allow: /lifestyle/apl-media/ for AI2Bot
- The Telegraph — Added Allow: /money/investing/fc-investment-trust/long-term-investing/ for AI2Bot
- The Telegraph — Added Allow: /rugby-union/grassroots-rugby-heroes/ for AI2Bot
- The Telegraph — Added Allow: /rugby-union/levelling-the-playing-field/ for AI2Bot
- The Telegraph — Added Allow: /sailgp/inside-f50-catamaran for AI2Bot
- The Telegraph — Added Allow: /seabourn/explore-the-mediterranean for AI2Bot
- The Telegraph — Added Allow: /sport/sailgp-championship/ for AI2Bot
- The Telegraph — Added Allow: /travel/active-family-holidays/ for AI2Bot
- The Telegraph — Added Allow: /travel/away-holidays/ for AI2Bot
- The Telegraph — Added Allow: /travel/barbados-holiday-guide/ for AI2Bot
- The Telegraph — Added Allow: /travel/british-airways-american-express/ for AI2Bot
- The Telegraph — Added Allow: /travel/canadian-affair-holidays/rocky-mountaineer-train-route/ for AI2Bot
- The Telegraph — Added Allow: /travel/croatia-holidays/ for AI2Bot
- The Telegraph — Added Allow: /travel/cruises/holland-america-line-luxury-holidays/ for AI2Bot
- The Telegraph — Added Allow: /travel/cruises/seabourn-luxury-cruises/ for AI2Bot
- The Telegraph — Added Allow: /travel/discover-america/ for AI2Bot
- The Telegraph — Added Allow: /travel/exclusiveoffers/ for AI2Bot
- The Telegraph — Added Allow: /travel/florida-guide/ for AI2Bot
- The Telegraph — Added Allow: /travel/fly-drive-holidays/ for AI2Bot
- The Telegraph — Added Allow: /travel/grenada-guide/ for AI2Bot
- The Telegraph — Added Allow: /travel/icons-of-the-sky/ for AI2Bot
- The Telegraph — Added Allow: /travel/indian-ocean-escapes/ for AI2Bot
- The Telegraph — Added Allow: /travel/indulge-in-ireland/ for AI2Bot
- The Telegraph — Added Allow: /travel/jamaica-holiday-guide/ for AI2Bot
- The Telegraph — Added Allow: /travel/japan-ninja-heritage/ for AI2Bot
- The Atlantic — Removed Disallow: /4624/TheAtlanticOnline/* for ChatGPT-User
- The Atlantic — Removed Disallow: /4624/TheAtlanticOnline/* for DuckAssistBot
- The Atlantic — Removed Disallow: /4624/TheAtlanticOnline/* for FacebookBot
- The Atlantic — Removed Disallow: /4624/TheAtlanticOnline/* for GPTBot
- The Atlantic — Removed Disallow: /4624/TheAtlanticOnline/* for OAI-SearchBot
Methodology. robots.txt files are fetched every 6 hours from the
primary domains of every monitored cohort (US & European companies,
governments and social networks). "Blocks" counts organisations that name an
AI crawler in their own User-agent group with
Disallow: /. 1203 of 1273 sites are reachable
and analysed. Browse cohorts on the datasets page and
trends charts. Full crawl health on the
status page.