The Wall

Every monitored site × AI crawlers and search engines, as of 2026-06-20 — 3 of 37 sites block at least one AI crawler by name.

All cohorts Top 1000 websites US 500 Nasdaq 100 AI labs Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC National indices Banks & fintech European companies Social Networks Governments Publishers Show all bots

blocked by name restricted by name via wildcard only allowed AI crawlers Search engines

Site	GPTBot	ChatGPT-User	ClaudeBot	anthropic-ai	CCBot	Google-Extended	Applebot-Extended	PerplexityBot	Bytespider	Amazonbot	meta-externalagent	FacebookBot	Googlebot	Bingbot	DuckDuckBot	Baiduspider	YandexBot	Applebot
🇺🇸 Midjourney
🇮🇱 AI21 Labs
🇺🇸 Together AI
🇨🇳 01.AI
🇩🇪 Aleph Alpha
🇺🇸 Anthropic
🇺🇸 Anysphere (Cursor)
🇺🇸 Cerebras
🇺🇸 Character.AI
🇨🇦 Cohere
🇺🇸 Contextual AI
🇺🇸 Databricks
🇺🇸 ElevenLabs
🇺🇸 Google DeepMind
🇺🇸 Groq
🇺🇸 Hugging Face
🇺🇸 Inflection AI
🇺🇸 Jasper
🇺🇸 LangChain
🇺🇸 Luma AI
🇺🇸 Meta AI
🇫🇷 Mistral AI
🇺🇸 OpenAI
🇺🇸 OpenRouter
🇺🇸 Perplexity
🇺🇸 Pika
🇺🇸 Pinecone
🇺🇸 Reka
🇺🇸 Replicate
🇺🇸 Runway
🇺🇸 Scale AI
🇬🇧 Stability AI
🇺🇸 Suno
🇺🇸 Udio
🇳🇱 Weaviate
🇺🇸 Writer
🇺🇸 xAI