The Wall

Every monitored site × AI crawlers and search engines, as of 2026-06-20 — 23 of 449 sites block at least one AI crawler by name.

All cohorts Top 1000 websites US 500 Nasdaq 100 AI labs Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC National indices Banks & fintech European companies Social Networks Governments Publishers Show fewer bots

blocked by name restricted by name via wildcard only allowed AI crawlers Search engines

Site	GPTBot	ChatGPT-User	OAI-SearchBot	ClaudeBot	Claude-User	Claude-SearchBot	anthropic-ai	Claude-Web	CCBot	Google-Extended	Applebot-Extended	PerplexityBot	Perplexity-User	Bytespider	Amazonbot	FacebookBot	meta-externalagent	meta-externalfetcher	cohere-ai	AI2Bot	Diffbot	omgili	YouBot	DuckAssistBot	MistralAI-User	PanguBot	Timpibot	Googlebot	Googlebot-Image	Googlebot-News	Bingbot	Slurp	DuckDuckBot	Baiduspider	YandexBot	Applebot	Sogou	SeznamBot	PetalBot
🇺🇸 Amazon
🇺🇸 Burlington Stores
🇺🇸 Parker Hannifin
🇺🇸 eBay
🇺🇸 Northrop Grumman
🇺🇸 Expeditors International
🇺🇸 Sempra
🇺🇸 Tenet Healthcare
🇺🇸 Arrow Electronics
🇺🇸 W.W. Grainger
🇺🇸 Electronic Arts
🇺🇸 Insight Enterprises
🇺🇸 Kimberly-Clark
🇺🇸 Prudential Financial
🇺🇸 Halliburton
🇺🇸 American Financial Group
🇺🇸 Chewy
🇺🇸 Diamondback Energy
🇺🇸 Lululemon Athletica
🇺🇸 PG&E
🇺🇸 ServiceNow
🇺🇸 Targa Resources
🇺🇸 Victoria's Secret
🇺🇸 3M
🇺🇸 Abbott Laboratories
🇺🇸 AbbVie
🇺🇸 ABM Industries
🇺🇸 Ace Hardware
🇺🇸 Adobe
🇺🇸 ADP
🇺🇸 Advance Auto Parts
🇺🇸 Advanced Micro Devices
🇺🇸 AECOM
🇺🇸 Aflac
🇺🇸 AGCO
🇺🇸 Airbnb
🇺🇸 Akamai Technologies
🇺🇸 Alaska Air Group
🇺🇸 Albemarle
🇺🇸 Albertsons
🇺🇸 Alcoa
🇺🇸 Allstate
🇺🇸 Ally Financial
🇺🇸 Alphabet
🇺🇸 Ameren
🇺🇸 American Airlines Group
🇺🇸 American Express
🇺🇸 American Family Insurance
🇺🇸 American International Group
🇺🇸 Ameriprise Financial
🇺🇸 Amgen
🇺🇸 Amphenol
🇺🇸 Analog Devices
🇺🇸 Apple
🇺🇸 Applied Materials
🇺🇸 Aramark
🇺🇸 Archer Daniels Midland
🇺🇸 Arista Networks
🇺🇸 Arthur J. Gallagher
🇺🇸 Asbury Automotive Group
🇺🇸 Assurant
🇺🇸 AT&T
🇺🇸 Auto-Owners Insurance
🇺🇸 AutoNation
🇺🇸 AutoZone
🇺🇸 Avery Dennison
🇺🇸 Avis Budget Group
🇺🇸 Avnet
🇺🇸 Ball
🇺🇸 Bank of America
🇺🇸 Bath & Body Works
🇺🇸 Becton Dickinson
🇺🇸 Berry Global
🇺🇸 Best Buy
🇺🇸 Biogen
🇺🇸 BlackRock
🇺🇸 Block
🇺🇸 BNY Mellon
🇺🇸 Boeing
🇺🇸 Booking Holdings
🇺🇸 Booz Allen Hamilton
🇺🇸 Boston Scientific
🇺🇸 Bristol-Myers Squibb
🇺🇸 Broadcom
🇺🇸 Builders FirstSource
🇺🇸 C.H. Robinson
🇺🇸 CACI International
🇺🇸 Caesars Entertainment
🇺🇸 Campbell's
🇺🇸 Capital One Financial
🇺🇸 Cardinal Health
🇺🇸 CarMax
🇺🇸 Carnival
🇺🇸 Carrier Global
🇺🇸 Caterpillar
🇺🇸 CBRE Group
🇺🇸 Celanese
🇺🇸 Cencora
🇺🇸 Centene
🇺🇸 CenterPoint Energy
🇺🇸 Charles Schwab
🇺🇸 Charter Communications
🇺🇸 Cheniere Energy
🇺🇸 Chevron
🇺🇸 CHS
🇺🇸 Chubb
🇺🇸 Cigna Group
🇺🇸 Cincinnati Financial
🇺🇸 Cintas
🇺🇸 Cisco Systems
🇺🇸 Citigroup
🇺🇸 Citizens Financial Group
🇺🇸 Cleveland-Cliffs
🇺🇸 Clorox
🇺🇸 CME Group
🇺🇸 Coca-Cola
🇺🇸 Cognizant
🇺🇸 Colgate-Palmolive
🇺🇸 Comcast
🇺🇸 Commercial Metals
🇺🇸 CommScope
🇺🇸 Community Health Systems
🇺🇸 Conagra Brands
🇺🇸 ConocoPhillips
🇺🇸 Consolidated Edison
🇺🇸 Constellation Brands
🇺🇸 Constellation Energy
🇺🇸 Corning
🇺🇸 Corteva
🇺🇸 Costco Wholesale
🇺🇸 Crown Holdings
🇺🇸 Cummins
🇺🇸 CVS Health
🇺🇸 Dairy Farmers of America
🇺🇸 Dana
🇺🇸 Danaher
🇺🇸 Darden Restaurants
🇺🇸 DaVita
🇺🇸 Deere
🇺🇸 Delek US Holdings
🇺🇸 Dell Technologies
🇺🇸 Delta Air Lines
🇺🇸 Dick's Sporting Goods
🇺🇸 Discover Financial Services
🇺🇸 Dollar General
🇺🇸 Dollar Tree
🇺🇸 Dominion Energy
🇺🇸 DoorDash
🇺🇸 Dover
🇺🇸 Dow
🇺🇸 DTE Energy
🇺🇸 Duke Energy
🇺🇸 DuPont
🇺🇸 DXC Technology
🇺🇸 Eastman Chemical
🇺🇸 Ecolab
🇺🇸 Edison International
🇺🇸 Edwards Lifesciences
🇺🇸 Elevance Health
🇺🇸 Eli Lilly
🇺🇸 EMCOR Group
🇺🇸 Emerson Electric
🇺🇸 Entergy
🇺🇸 Equitable Holdings
🇺🇸 Erie Insurance
🇺🇸 Estee Lauder
🇺🇸 Exelon
🇺🇸 Expedia Group
🇺🇸 ExxonMobil
🇺🇸 Fannie Mae
🇺🇸 Fastenal
🇺🇸 FedEx
🇺🇸 Ferguson
🇺🇸 Fidelity National Information Services
🇺🇸 Fifth Third Bancorp
🇺🇸 First American Financial
🇺🇸 First Citizens BancShares
🇺🇸 FirstEnergy
🇺🇸 Fiserv
🇺🇸 Fluor
🇺🇸 Foot Locker
🇺🇸 Ford Motor
🇺🇸 Fortive
🇺🇸 Fox
🇺🇸 Freddie Mac
🇺🇸 Freeport-McMoRan
🇺🇸 Frontier Communications
🇺🇸 Gap
🇺🇸 GE HealthCare
🇺🇸 GE Vernova
🇺🇸 General Dynamics
🇺🇸 General Electric
🇺🇸 General Mills
🇺🇸 General Motors
🇺🇸 Gilead Sciences
🇺🇸 Global Payments
🇺🇸 GoDaddy
🇺🇸 Goldman Sachs
🇺🇸 Goodyear Tire & Rubber
🇺🇸 Graphic Packaging
🇺🇸 GXO Logistics
🇺🇸 Harley-Davidson
🇺🇸 Hartford Financial Services
🇺🇸 HCA Healthcare
🇺🇸 Hertz Global Holdings
🇺🇸 Hewlett Packard Enterprise
🇺🇸 HF Sinclair
🇺🇸 Hilton Worldwide
🇺🇸 Home Depot
🇺🇸 Honeywell International
🇺🇸 Hormel Foods
🇺🇸 Howmet Aerospace
🇺🇸 HP
🇺🇸 Humana
🇺🇸 Huntington Bancshares
🇺🇸 Huntington Ingalls Industries
🇺🇸 IBM
🇺🇸 Illinois Tool Works
🇺🇸 Illumina
🇺🇸 Ingram Micro
🇺🇸 Ingredion
🇺🇸 Intel
🇺🇸 Intercontinental Exchange
🇺🇸 International Flavors & Fragrances
🇺🇸 International Paper
🇺🇸 Interpublic Group
🇺🇸 Intuit
🇺🇸 Intuitive Surgical
🇺🇸 IQVIA Holdings
🇺🇸 J.B. Hunt Transport
🇺🇸 J.M. Smucker
🇺🇸 Jacobs Solutions
🇺🇸 JetBlue Airways
🇺🇸 Johnson & Johnson
🇺🇸 Jones Financial
🇺🇸 Jones Lang LaSalle
🇺🇸 JPMorgan Chase
🇺🇸 Juniper Networks
🇺🇸 KBR
🇺🇸 Keurig Dr Pepper
🇺🇸 KeyCorp
🇺🇸 Keysight Technologies
🇺🇸 Kinder Morgan
🇺🇸 KLA
🇺🇸 Kohl's
🇺🇸 Kroger
🇺🇸 Kyndryl
🇺🇸 L3Harris Technologies
🇺🇸 Labcorp
🇺🇸 Lam Research
🇺🇸 Land O'Lakes
🇺🇸 Las Vegas Sands
🇺🇸 Lear
🇺🇸 Leidos
🇺🇸 Lennar
🇺🇸 Levi Strauss
🇺🇸 Liberty Mutual Insurance
🇺🇸 Lincoln Financial
🇺🇸 Lithia Motors
🇺🇸 Live Nation Entertainment
🇺🇸 LKQ
🇺🇸 Lockheed Martin
🇺🇸 Loews
🇺🇸 Love's Travel Stops
🇺🇸 Lowe's
🇺🇸 Lucid Group
🇺🇸 Lumen Technologies
🇺🇸 Lyft
🇺🇸 LyondellBasell
🇺🇸 M&T Bank
🇺🇸 Macy's
🇺🇸 ManpowerGroup
🇺🇸 Markel Group
🇺🇸 Marriott International
🇺🇸 Marsh McLennan
🇺🇸 Martin Marietta Materials
🇺🇸 Masco
🇺🇸 MassMutual
🇺🇸 MasTec
🇺🇸 Mastercard
🇺🇸 McDonald's
🇺🇸 McKesson
🇺🇸 Merck
🇺🇸 Meta Platforms
🇺🇸 MetLife
🇺🇸 MGM Resorts International
🇺🇸 Micron Technology
🇺🇸 Microsoft
🇺🇸 Moderna
🇺🇸 Mohawk Industries
🇺🇸 Molina Healthcare
🇺🇸 Molson Coors
🇺🇸 Mondelez International
🇺🇸 Monster Beverage
🇺🇸 Moody's
🇺🇸 Morgan Stanley
🇺🇸 Mosaic
🇺🇸 Motorola Solutions
🇺🇸 Murphy USA
🇺🇸 Nationwide
🇺🇸 NetApp
🇺🇸 Netflix
🇺🇸 New York Life Insurance
🇺🇸 Newell Brands
🇺🇸 Newmont
🇺🇸 News Corp
🇺🇸 NextEra Energy
🇺🇸 Nike
🇺🇸 Nordstrom
🇺🇸 Northwestern Mutual
🇺🇸 Norwegian Cruise Line
🇺🇸 NOV
🇺🇸 NRG Energy
🇺🇸 Nvidia
🇺🇸 O'Reilly Automotive
🇺🇸 Occidental Petroleum
🇺🇸 Old Republic International
🇺🇸 Omnicom Group
🇺🇸 Oneok
🇺🇸 Onsemi
🇺🇸 Oracle
🇺🇸 Oshkosh
🇺🇸 Otis Worldwide
🇺🇸 Owens & Minor
🇺🇸 Owens Corning
🇺🇸 Packaging Corp of America
🇺🇸 Palo Alto Networks
🇺🇸 Paramount Global
🇺🇸 PayPal Holdings
🇺🇸 PBF Energy
🇺🇸 Penske Automotive Group
🇺🇸 PepsiCo
🇺🇸 Performance Food Group
🇺🇸 Pfizer
🇺🇸 Philip Morris International
🇺🇸 Phillips 66
🇺🇸 Pilgrim's Pride
🇺🇸 Pilot Company
🇺🇸 Plains GP Holdings
🇺🇸 PNC Financial Services
🇺🇸 Polaris
🇺🇸 Post Holdings
🇺🇸 PPG Industries
🇺🇸 Principal Financial
🇺🇸 Procter & Gamble
🇺🇸 Progressive
🇺🇸 Prologis
🇺🇸 Public Service Enterprise Group
🇺🇸 Publix Super Markets
🇺🇸 PulteGroup
🇺🇸 Pure Storage
🇺🇸 PVH
🇺🇸 Qualcomm
🇺🇸 Quanta Services
🇺🇸 Quest Diagnostics
🇺🇸 Ralph Lauren
🇺🇸 Raymond James Financial
🇺🇸 Regeneron Pharmaceuticals
🇺🇸 Regions Financial
🇺🇸 Reinsurance Group of America
🇺🇸 Reliance Steel & Aluminum
🇺🇸 Republic Services
🇺🇸 Rivian Automotive
🇺🇸 Rockwell Automation
🇺🇸 Ross Stores
🇺🇸 Royal Caribbean Group
🇺🇸 RTX
🇺🇸 Ryder System
🇺🇸 S&P Global
🇺🇸 Salesforce
🇺🇸 Science Applications International
🇺🇸 Seaboard
🇺🇸 Seagate Technology
🇺🇸 Sherwin-Williams
🇺🇸 Sirius XM
🇺🇸 Skechers
🇺🇸 SLB
🇺🇸 Snowflake
🇺🇸 Sonic Automotive
🇺🇸 Southern Company
🇺🇸 Southwest Airlines
🇺🇸 Stanley Black & Decker
🇺🇸 Starbucks
🇺🇸 State Farm Insurance
🇺🇸 State Street
🇺🇸 Steel Dynamics
🇺🇸 StoneX Group
🇺🇸 Stryker
🇺🇸 Super Micro Computer
🇺🇸 Synchrony Financial
🇺🇸 Synopsys
🇺🇸 Sysco
🇺🇸 Take-Two Interactive
🇺🇸 Tapestry
🇺🇸 Target
🇺🇸 Taylor Morrison
🇺🇸 Tesla
🇺🇸 Texas Instruments
🇺🇸 Textron
🇺🇸 Thermo Fisher Scientific
🇺🇸 Thor Industries
🇺🇸 TIAA
🇺🇸 Toll Brothers
🇺🇸 Tractor Supply
🇺🇸 TransDigm Group
🇺🇸 Travelers
🇺🇸 Truist Financial
🇺🇸 Tyson Foods
🇺🇸 U.S. Bancorp
🇺🇸 U.S. Steel
🇺🇸 Uber Technologies
🇺🇸 Union Pacific
🇺🇸 United Airlines Holdings
🇺🇸 United Parcel Service
🇺🇸 United Rentals
🇺🇸 UnitedHealth Group
🇺🇸 Universal Health Services
🇺🇸 Unum Group
🇺🇸 US Foods
🇺🇸 USAA
🇺🇸 Valero Energy
🇺🇸 Verizon Communications
🇺🇸 Vertex Pharmaceuticals
🇺🇸 Viatris
🇺🇸 Visa
🇺🇸 Vistra
🇺🇸 Voya Financial
🇺🇸 Vulcan Materials
🇺🇸 W.R. Berkley
🇺🇸 Wabtec
🇺🇸 Walgreens Boots Alliance
🇺🇸 Walmart
🇺🇸 Walt Disney
🇺🇸 Warner Bros. Discovery
🇺🇸 Waste Management
🇺🇸 Watsco
🇺🇸 Wayfair
🇺🇸 Wells Fargo
🇺🇸 Western Digital
🇺🇸 Westlake
🇺🇸 Weyerhaeuser
🇺🇸 Williams
🇺🇸 Workday
🇺🇸 World Kinect
🇺🇸 XPO
🇺🇸 Xylem
🇺🇸 Zebra Technologies
🇺🇸 Zimmer Biomet
🇺🇸 Zoetis