The Wall
Every monitored site × AI crawlers and search engines, as of 2026-06-20 — 13 of 25 sites block at least one AI crawler by name.
All cohorts
Top 1000 websites
US 500
Nasdaq 100
AI labs
Cybersecurity
SEO & AI search
Creative rights-holders
Knowledge & UGC
National indices
Banks & fintech
European companies
Social Networks
Governments
Publishers
Show fewer bots
blocked by name
restricted by name
via wildcard only
allowed
AI crawlers
Search engines