The Wall
Every monitored site × AI crawlers and search engines, as of 2020-09-30 — 5 of 20 sites block at least one AI crawler by name.
All cohorts
Top 1000 websites
US 500
Nasdaq 100
AI models
Cybersecurity
SEO & AI search
Creative rights-holders
Knowledge & UGC
Academic & open research
Universities
National indices
Banks & fintech
E-commerce
European companies
Social Networks
Governments
Publishers
Show all bots
blocked by name
restricted by name
via wildcard only
allowed
AI crawlers
Search engines