The Wall

Every monitored site × AI crawlers and search engines, as of 2025-02-22 — 4 of 53 sites block at least one AI crawler by name.

All cohorts Top 1000 websites US 500 Nasdaq 100 AI models Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC Academic & open research Universities National indices Banks & fintech E-commerce European companies Social Networks Governments Publishers Show fewer bots

blocked by name restricted by name via wildcard only allowed AI crawlers Search engines

Site	GPTBot	ChatGPT-User	OAI-SearchBot	ClaudeBot	Claude-User	Claude-SearchBot	anthropic-ai	Claude-Web	CCBot	Google-Extended	Applebot-Extended	PerplexityBot	Perplexity-User	Bytespider	Amazonbot	FacebookBot	meta-externalagent	meta-externalfetcher	cohere-ai	AI2Bot	Diffbot	omgili	YouBot	DuckAssistBot	MistralAI-User	PanguBot	Timpibot	Googlebot	Googlebot-Image	Googlebot-News	Bingbot	Slurp	DuckDuckBot	Baiduspider	YandexBot	Applebot	Sogou	SeznamBot	PetalBot
🇺🇸 United States (USA.gov)
🇨🇦 Canada (Government of Canada)
🇬🇧 United Kingdom (GOV.UK)
🇫🇷 France (Gouvernement)
🇩🇪 Germany (Bundesregierung)
🇮🇹 Italy (Governo)
🇪🇸 Spain (La Moncloa)
🇳🇱 Netherlands (Government.nl)
🇧🇪 Belgium (Belgium.be)
🇮🇪 Ireland (Gov.ie)
🇵🇹 Portugal (Governo)
🇸🇪 Sweden (Regeringen)
🇳🇴 Norway (Regjeringen)
🇩🇰 Denmark (Statsministeriet)
🇫🇮 Finland (Valtioneuvosto)
🇮🇸 Iceland (Government.is)
🇨🇭 Switzerland (Admin.ch)
🇦🇹 Austria (Bundeskanzleramt)
🇱🇺 Luxembourg (Gouvernement)
🇵🇱 Poland (Gov.pl)
🇨🇿 Czech Republic (Vlada)
🇸🇮 Slovenia (Gov.si)
🇧🇬 Bulgaria (Government)
🇬🇷 Greece (Government)
🇨🇾 Cyprus (Gov.cy)
🇱🇻 Latvia (Ministru kabinets)
🇱🇹 Lithuania (LRV)
🇭🇺 Hungary (Kormany)
🇦🇱 Albania (Kryeministria)
🇲🇰 North Macedonia (Vlada)
🇲🇪 Montenegro (Gov.me)
🇲🇩 Moldova (Gov.md)
🇺🇦 Ukraine (KMU)
🇦🇲 Armenia (Gov.am)
🇰🇷 South Korea (Korea.kr)
🇮🇳 India (India.gov.in)
🇦🇺 Australia (my.gov.au)
🇲🇾 Malaysia (Malaysia.gov.my)
🇧🇼 Botswana (Gov.bw)
🇬🇭 Ghana (Ghana.gov.gh)
🇨🇻 Cape Verde (Governo)
🇧🇷 Brazil (Gov.br)
🇦🇷 Argentina (Argentina.gob.ar)
🇺🇾 Uruguay (Gub.uy)
🇨🇷 Costa Rica (Presidencia)
🇲🇽 Mexico (Gob.mx)
🇨🇴 Colombia (Presidencia)
🇵🇪 Peru (Gob.pe)
🇵🇦 Panama (Presidencia)
🇩🇴 Dominican Republic (Presidencia)
🇯🇲 Jamaica (Office of the PM)
🇬🇾 Guyana (Office of the President)
🇸🇷 Suriname (Gov.sr)