The Wall

Every monitored site × AI crawlers and search engines, as of 2026-06-20 — 16 of 39 sites block at least one AI crawler by name.

All cohorts Top 1000 websites US 500 Nasdaq 100 AI labs Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC National indices Banks & fintech European companies Social Networks Governments Publishers Show fewer bots

blocked by name restricted by name via wildcard only allowed AI crawlers Search engines

Site	GPTBot	ChatGPT-User	OAI-SearchBot	ClaudeBot	Claude-User	Claude-SearchBot	anthropic-ai	Claude-Web	CCBot	Google-Extended	Applebot-Extended	PerplexityBot	Perplexity-User	Bytespider	Amazonbot	FacebookBot	meta-externalagent	meta-externalfetcher	cohere-ai	AI2Bot	Diffbot	omgili	YouBot	DuckAssistBot	MistralAI-User	PanguBot	Timpibot	Googlebot	Googlebot-Image	Googlebot-News	Bingbot	Slurp	DuckDuckBot	Baiduspider	YandexBot	Applebot	Sogou	SeznamBot	PetalBot
🇺🇸 wikiHow
🇺🇸 Genius
🇺🇸 Behance
🇬🇧 Letterboxd
🇺🇸 Figma Community
🇺🇸 Quora
🇺🇸 Tumblr
🇺🇸 Medium
🇺🇸 Scribd
🇺🇸 SlideShare
🇺🇸 Archive of Our Own
🇺🇸 Goodreads
🇺🇸 Coursera
🇺🇸 Khan Academy
🇺🇸 Notion
🇨🇦 Wattpad
🇺🇸 DEV Community
🇺🇸 Fandom
🇺🇸 Giphy
🇺🇸 GitHub
🇺🇸 GitLab
🇺🇸 Hacker News
🇺🇸 Imgur
🇺🇸 Instructables
🇺🇸 Internet Archive
🇺🇸 MDN Web Docs
🇺🇸 npm
🇺🇸 Project Gutenberg
🇺🇸 PyPI
🇺🇸 Read the Docs
🇺🇸 Reddit
🇺🇸 Stack Exchange
🇺🇸 Stack Overflow
🇺🇸 Substack
🇳🇴 W3Schools
🇺🇸 Wikidata
🇺🇸 Wikimedia Commons
🇺🇸 Wikipedia
🇺🇸 Wiktionary