The Wall

Every monitored site × AI crawlers and search engines, as of 2026-06-20 — 5 of 154 sites block at least one AI crawler by name.

All cohorts Top 1000 websites US 500 Nasdaq 100 AI labs Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC National indices Banks & fintech European companies Social Networks Governments Publishers Show fewer bots

blocked by name restricted by name via wildcard only allowed AI crawlers Search engines

Site	GPTBot	ChatGPT-User	OAI-SearchBot	ClaudeBot	Claude-User	Claude-SearchBot	anthropic-ai	Claude-Web	CCBot	Google-Extended	Applebot-Extended	PerplexityBot	Perplexity-User	Bytespider	Amazonbot	FacebookBot	meta-externalagent	meta-externalfetcher	cohere-ai	AI2Bot	Diffbot	omgili	YouBot	DuckAssistBot	MistralAI-User	PanguBot	Timpibot	Googlebot	Googlebot-Image	Googlebot-News	Bingbot	Slurp	DuckDuckBot	Baiduspider	YandexBot	Applebot	Sogou	SeznamBot	PetalBot
🇩🇪 Lufthansa Group
🇱🇺 ArcelorMittal
🇪🇸 El Corte Ingles
🇬🇧 WPP
🇫🇷 Airbus
🇩🇰 A.P. Moller-Maersk
🇧🇪 AB InBev
🇮🇪 Accenture
🇫🇷 Accor
🇪🇸 ACS Group
🇩🇪 Adidas
🇳🇱 Adyen
🇳🇱 Ahold Delhaize
🇫🇷 Air France-KLM
🇫🇷 Air Liquide
🇳🇱 AkzoNobel
🇩🇪 Allianz
🇳🇱 ASML
🇬🇧 AstraZeneca
🇬🇧 Aviva
🇫🇷 AXA
🇩🇪 Axel Springer
🇬🇧 BAE Systems
🇪🇸 Banco Santander
🇬🇧 Barclays
🇩🇪 BASF
🇩🇪 Bayer
🇩🇪 Beiersdorf
🇩🇪 Bertelsmann
🇩🇪 BMW Group
🇫🇷 BNP Paribas
🇩🇪 Boehringer Ingelheim
🇳🇱 Booking.com
🇩🇪 Bosch
🇫🇷 Bouygues
🇬🇧 BP
🇬🇧 BT Group
🇫🇷 Capgemini
🇫🇷 Carrefour
🇫🇷 CMA CGM
🇬🇧 Compass Group
🇩🇪 Continental
🇫🇷 Credit Agricole
🇮🇪 CRH
🇫🇷 Danone
🇫🇷 Decathlon
🇩🇪 Deutsche Bank
🇩🇪 Deutsche Telekom
🇩🇪 DHL Group
🇬🇧 Diageo
🇩🇰 DSV
🇩🇪 E.ON
🇫🇷 EDF
🇮🇹 Enel
🇫🇷 Engie
🇮🇹 Eni
🇸🇪 Ericsson
🇫🇷 EssilorLuxottica
🇩🇪 Evonik
🇮🇹 Ferrari
🇪🇸 Ferrovial
🇩🇪 Fresenius
🇮🇹 Generali
🇨🇭 Glencore
🇬🇧 GSK
🇸🇪 H&M Group
🇬🇧 Haleon
🇩🇪 Hapag-Lloyd
🇩🇪 Heidelberg Materials
🇳🇱 Heineken
🇩🇪 Henkel
🇫🇷 Hermes
🇨🇭 Holcim
🇬🇧 HSBC
🇪🇸 IAG
🇪🇸 Iberdrola
🇩🇪 Infineon
🇳🇱 ING Group
🇳🇱 Ingka Group (IKEA)
🇮🇹 Intesa Sanpaolo
🇫🇷 Kering
🇳🇱 KPN
🇨🇭 Kuehne+Nagel
🇫🇷 L'Oreal
🇬🇧 Legal & General
🇮🇹 Leonardo
🇫🇷 Leroy Merlin (Adeo)
🇬🇧 Lloyds Banking Group
🇫🇷 LVMH
🇪🇸 Mercadona
🇩🇪 Mercedes-Benz Group
🇩🇪 Merck KGaA
🇩🇪 Metro AG
🇫🇷 Michelin
🇩🇪 Munich Re
🇪🇸 Naturgy
🇬🇧 NatWest Group
🇨🇭 Nestle
🇫🇮 Nokia
🇫🇮 Nordea
🇨🇭 Novartis
🇩🇰 Novo Nordisk
🇦🇹 OMV
🇫🇷 Orange
🇵🇱 Orlen
🇫🇷 Pernod Ricard
🇳🇱 Philips
🇩🇪 Porsche
🇳🇱 Prosus
🇫🇷 Publicis Groupe
🇬🇧 Reckitt
🇬🇧 RELX
🇫🇷 Renault Group
🇪🇸 Repsol
🇩🇪 REWE Group
🇨🇭 Richemont
🇨🇭 Roche
🇬🇧 Rolls-Royce
🇩🇪 RWE
🇮🇪 Ryanair
🇫🇷 Safran
🇫🇷 Saint-Gobain
🇫🇷 Sanofi
🇩🇪 SAP
🇫🇷 Schneider Electric
🇩🇪 Schwarz Group
🇬🇧 Shell
🇩🇪 Siemens
🇸🇪 Skanska
🇫🇷 Societe Generale
🇫🇷 Sodexo
🇧🇪 Solvay
🇸🇪 Spotify
🇳🇱 Stellantis
🇨🇭 STMicroelectronics
🇨🇭 Swiss Re
🇨🇭 Swisscom
🇪🇸 Telefonica
🇸🇪 Telia
🇬🇧 Tesco
🇫🇷 Thales
🇩🇪 Thyssenkrupp
🇫🇷 TotalEnergies
🇨🇭 UBS
🇮🇹 UniCredit
🇬🇧 Unilever
🇸🇪 Vattenfall
🇫🇷 Veolia
🇫🇷 Vinci
🇬🇧 Vodafone
🇩🇪 Volkswagen Group
🇸🇪 Volvo Cars
🇸🇪 Volvo Group
🇨🇭 Zurich Insurance