The Wall

Every monitored site × AI crawlers and search engines, as of 2026-06-20 — 4 of 53 sites block at least one AI crawler by name.

All cohorts Top 1000 websites US 500 Nasdaq 100 AI labs Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC National indices Banks & fintech European companies Social Networks Governments Publishers Show fewer bots

blocked by name restricted by name via wildcard only allowed AI crawlers Search engines

Site	GPTBot	ChatGPT-User	OAI-SearchBot	ClaudeBot	Claude-User	Claude-SearchBot	anthropic-ai	Claude-Web	CCBot	Google-Extended	Applebot-Extended	PerplexityBot	Perplexity-User	Bytespider	Amazonbot	FacebookBot	meta-externalagent	meta-externalfetcher	cohere-ai	AI2Bot	Diffbot	omgili	YouBot	DuckAssistBot	MistralAI-User	PanguBot	Timpibot	Googlebot	Googlebot-Image	Googlebot-News	Bingbot	Slurp	DuckDuckBot	Baiduspider	YandexBot	Applebot	Sogou	SeznamBot	PetalBot
🇺🇸 Proofpoint
🇺🇸 Arkose Labs
🇺🇸 Barracuda
🇺🇸 Fortinet
🇺🇸 Akamai Technologies
🇺🇸 Arctic Wolf
🇨🇿 Avast
🇷🇴 Bitdefender
🇺🇸 Broadcom
🇮🇱 Check Point
🇺🇸 Cisco Systems
🇺🇸 Cloudflare
🇺🇸 Cohesity
🇺🇸 CrowdStrike
🇺🇸 CyberArk
🇬🇧 Darktrace
🇺🇸 Dragos
🇸🇰 ESET
🇺🇸 ExtraHop
🇺🇸 F5
🇺🇸 Fortra
🇺🇸 Gen Digital
🇺🇸 Human Security
🇺🇸 Illumio
🇺🇸 Imperva
🇦🇺 Kasada
🇨🇭 Kaspersky
🇺🇸 Lookout
🇺🇸 Malwarebytes
🇺🇸 McAfee
🇺🇸 Mimecast
🇺🇸 Netskope
🇺🇸 Okta
🇺🇸 OneTrust
🇺🇸 Orca Security
🇺🇸 Palo Alto Networks
🇺🇸 Qualys
🇺🇸 Rapid7
🇺🇸 Recorded Future
🇺🇸 Rubrik
🇺🇸 SentinelOne
🇺🇸 Snyk
🇬🇧 Sophos
🇺🇸 Splunk
🇺🇸 Tanium
🇺🇸 Tenable
🇯🇵 Trend Micro
🇺🇸 Varonis
🇺🇸 Vectra AI
🇫🇮 WithSecure
🇺🇸 Wiz
🇺🇸 Zimperium
🇺🇸 Zscaler