The Wall

Every monitored site × AI crawlers and search engines, as of 2020-09-03 — 16 of 39 sites block at least one AI crawler by name.

All cohorts Top 1000 websites US 500 Nasdaq 100 AI models Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC Academic & open research Universities National indices Banks & fintech E-commerce European companies Social Networks Governments Publishers Show all bots

blocked by name restricted by name via wildcard only allowed AI crawlers Search engines

Site	GPTBot	ChatGPT-User	ClaudeBot	anthropic-ai	CCBot	Google-Extended	Applebot-Extended	PerplexityBot	Bytespider	Amazonbot	meta-externalagent	Meta-WebIndexer	FacebookBot	Googlebot	Bingbot	DuckDuckBot	Baiduspider	YandexBot	Applebot
🇺🇸 Reddit
🇺🇸 Tumblr
🇺🇸 Medium
🇺🇸 Quora
🇺🇸 Substack
🇺🇸 Coursera
🇳🇴 W3Schools
🇺🇸 MDN Web Docs
🇺🇸 Wikidata
🇨🇦 Wattpad
🇺🇸 Archive of Our Own
🇺🇸 Imgur
🇺🇸 Giphy
🇺🇸 Notion
🇺🇸 Figma Community
🇺🇸 Behance
🇺🇸 Wikipedia
🇺🇸 Stack Overflow
🇺🇸 Stack Exchange
🇺🇸 GitHub
🇺🇸 GitLab
🇺🇸 Fandom
🇺🇸 wikiHow
🇺🇸 Goodreads
🇺🇸 Genius
🇺🇸 DEV Community
🇺🇸 Hacker News
🇺🇸 Wikimedia Commons
🇺🇸 Wiktionary
🇺🇸 Internet Archive
🇺🇸 Project Gutenberg
🇺🇸 Scribd
🇺🇸 SlideShare
🇺🇸 Read the Docs
🇺🇸 npm
🇺🇸 PyPI
🇬🇧 Letterboxd
🇺🇸 Instructables
🇺🇸 Khan Academy