The Wall

Every monitored site × AI crawlers and search engines, as of 2026-06-20 — 50 of 63 sites block at least one AI crawler by name.

All cohorts Top 1000 websites US 500 Nasdaq 100 AI labs Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC National indices Banks & fintech European companies Social Networks Governments Publishers Show all bots

blocked by name restricted by name via wildcard only allowed AI crawlers Search engines

Site	GPTBot	ChatGPT-User	ClaudeBot	anthropic-ai	CCBot	Google-Extended	Applebot-Extended	PerplexityBot	Bytespider	Amazonbot	meta-externalagent	FacebookBot	Googlebot	Bingbot	DuckDuckBot	Baiduspider	YandexBot	Applebot
🇬🇧 Daily Mail
🇺🇸 HuffPost
🇺🇸 NBC News
🇯🇵 Nikkei
🇺🇸 Politico
🇯🇵 The Asahi Shimbun
🇦🇺 The Sydney Morning Herald
🇬🇧 The Telegraph
🇨🇦 Toronto Star
🇺🇸 USA Today
🇬🇧 BBC
🇺🇸 Bloomberg
🇺🇸 BuzzFeed
🇺🇸 CNN
🇳🇱 De Telegraaf
🇬🇧 Sky News
🇺🇸 The Hill
🇺🇸 The New York Times
🇺🇸 Ars Technica
🇩🇪 Die Zeit
🇺🇸 Forbes
🇺🇸 Newsweek
🇺🇸 NPR
🇨🇦 The Globe and Mail
🇺🇸 The Verge
🇺🇸 Vox
🇩🇪 Der Spiegel
🇬🇧 Financial Times
🇺🇸 Slate
🇺🇸 TechCrunch
🇺🇸 The Atlantic
🇬🇧 The Guardian
🇺🇸 The Washington Post
🇮🇹 Corriere della Sera
🇫🇷 Le Figaro
🇫🇷 Le Monde
🇬🇧 The Economist
🇺🇸 The New Yorker
🇺🇸 Wired
🇺🇸 ABC News
🇺🇸 The Associated Press
🇺🇸 The Intercept
🇶🇦 Al Jazeera
🇺🇸 Axios
🇩🇪 Bild
🇺🇸 Business Insider
🇪🇸 El País
🇺🇸 Los Angeles Times
🇺🇸 CBS News
🇮🇳 The Times of India
🇫🇷 Agence France-Presse
🇩🇪 Axel Springer
🇺🇸 Condé Nast
🇺🇸 Fox News
🇺🇸 Gannett
🇺🇸 News Corp
🇺🇸 ProPublica
🇺🇸 Reuters
🇭🇰 South China Morning Post
🇬🇧 The Independent
🇬🇧 The Times
🇺🇸 The Wall Street Journal
🇺🇸 Time