The Wall

Every monitored site × AI crawlers and search engines, as of 2025-08-04 — 2 of 33 sites block at least one AI crawler by name.

All cohorts Top 1000 websites US 500 Nasdaq 100 AI models Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC Academic & open research Universities National indices Banks & fintech E-commerce European companies Social Networks Governments Publishers Show fewer bots

blocked by name restricted by name via wildcard only allowed AI crawlers Search engines

Site	GPTBot	ChatGPT-User	OAI-SearchBot	ClaudeBot	Claude-User	Claude-SearchBot	anthropic-ai	Claude-Web	CCBot	Google-Extended	Applebot-Extended	PerplexityBot	Perplexity-User	Bytespider	Amazonbot	FacebookBot	meta-externalagent	meta-externalfetcher	cohere-ai	AI2Bot	Diffbot	omgili	YouBot	DuckAssistBot	MistralAI-User	PanguBot	Timpibot	Googlebot	Googlebot-Image	Googlebot-News	Bingbot	Slurp	DuckDuckBot	Baiduspider	YandexBot	Applebot	Sogou	SeznamBot	PetalBot
🇺🇸 Massachusetts Institute of Technology
🇬🇧 University of Cambridge
🇺🇸 Harvard University
🇺🇸 Stanford University
🇺🇸 University of California Berkeley
🇺🇸 California Institute of Technology
🇺🇸 Princeton University
🇺🇸 Yale University
🇺🇸 University of Chicago
🇺🇸 University of Pennsylvania
🇺🇸 Cornell University
🇺🇸 University of California Los Angeles
🇺🇸 University of Michigan
🇬🇧 Imperial College London
🇬🇧 University College London
🇬🇧 University of Edinburgh
🇨🇭 EPFL
🇩🇪 Technical University of Munich
🇩🇪 Ludwig Maximilian University of Munich
🇫🇷 Sorbonne University
🇳🇱 University of Amsterdam
🇸🇪 KTH Royal Institute of Technology
🇯🇵 University of Tokyo
🇯🇵 Kyoto University
🇸🇬 Nanyang Technological University
🇦🇺 University of Sydney
🇮🇳 Indian Institute of Technology Bombay
🇮🇳 Indian Institute of Science
🇨🇦 University of Toronto
🇨🇦 University of British Columbia
🇨🇦 McGill University
🇮🇱 Tel Aviv University
🇿🇦 University of Cape Town