The Wall

Every monitored site × AI crawlers and search engines, as of 2026-06-20 — 3 of 52 sites block at least one AI crawler by name.

All cohorts Top 1000 websites US 500 Nasdaq 100 AI labs Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC National indices Banks & fintech European companies Social Networks Governments Publishers Show fewer bots

blocked by name restricted by name via wildcard only allowed AI crawlers Search engines

Site	GPTBot	ChatGPT-User	OAI-SearchBot	ClaudeBot	Claude-User	Claude-SearchBot	anthropic-ai	Claude-Web	CCBot	Google-Extended	Applebot-Extended	PerplexityBot	Perplexity-User	Bytespider	Amazonbot	FacebookBot	meta-externalagent	meta-externalfetcher	cohere-ai	AI2Bot	Diffbot	omgili	YouBot	DuckAssistBot	MistralAI-User	PanguBot	Timpibot	Googlebot	Googlebot-Image	Googlebot-News	Bingbot	Slurp	DuckDuckBot	Baiduspider	YandexBot	Applebot	Sogou	SeznamBot	PetalBot
🇨🇷 Costa Rica (Presidencia)
🇬🇾 Guyana (Office of the President)
🇮🇸 Iceland (Government.is)
🇦🇱 Albania (Kryeministria)
🇦🇷 Argentina (Argentina.gob.ar)
🇦🇲 Armenia (Gov.am)
🇦🇺 Australia (my.gov.au)
🇦🇹 Austria (Bundeskanzleramt)
🇧🇪 Belgium (Belgium.be)
🇧🇼 Botswana (Gov.bw)
🇧🇷 Brazil (Gov.br)
🇧🇬 Bulgaria (Government)
🇨🇦 Canada (Government of Canada)
🇨🇻 Cape Verde (Governo)
🇨🇴 Colombia (Presidencia)
🇨🇾 Cyprus (Gov.cy)
🇨🇿 Czech Republic (Vlada)
🇩🇰 Denmark (Statsministeriet)
🇩🇴 Dominican Republic (Presidencia)
🇫🇷 France (Gouvernement)
🇩🇪 Germany (Bundesregierung)
🇬🇭 Ghana (Ghana.gov.gh)
🇬🇷 Greece (Government)
🇭🇺 Hungary (Kormany)
🇮🇳 India (India.gov.in)
🇮🇪 Ireland (Gov.ie)
🇮🇹 Italy (Governo)
🇯🇲 Jamaica (Office of the PM)
🇱🇻 Latvia (Ministru kabinets)
🇱🇹 Lithuania (LRV)
🇱🇺 Luxembourg (Gouvernement)
🇲🇾 Malaysia (Malaysia.gov.my)
🇲🇽 Mexico (Gob.mx)
🇲🇩 Moldova (Gov.md)
🇲🇪 Montenegro (Gov.me)
🇳🇱 Netherlands (Government.nl)
🇲🇰 North Macedonia (Vlada)
🇳🇴 Norway (Regjeringen)
🇵🇦 Panama (Presidencia)
🇵🇪 Peru (Gob.pe)
🇵🇱 Poland (Gov.pl)
🇵🇹 Portugal (Governo)
🇸🇮 Slovenia (Gov.si)
🇰🇷 South Korea (Korea.kr)
🇪🇸 Spain (La Moncloa)
🇸🇷 Suriname (Gov.sr)
🇸🇪 Sweden (Regeringen)
🇨🇭 Switzerland (Admin.ch)
🇺🇦 Ukraine (KMU)
🇬🇧 United Kingdom (GOV.UK)
🇺🇸 United States (USA.gov)
🇺🇾 Uruguay (Gub.uy)