European companies
The archive — every monitored site, its robots.txt history and AI-crawler stance.
All sites
Top 1000 websites
US 500
Nasdaq 100
AI models
Cybersecurity
SEO & AI search
Creative rights-holders
Knowledge & UGC
Academic & open research
Universities
National indices
Banks & fintech
E-commerce
European companies
Social Networks
Governments
Publishers
| # | Company | Domain | Sector | robots.txt | AI bots blocked | Snapshots | Changes | Last change | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 🇩🇪 | Volkswagen Group |
|
Motor Vehicles | present | 0 | 1 | 0 | — |
| 2 | 🇬🇧 | Shell |
|
Energy | present | 0 | 14 | 13 | 2d ago |
| 3 | 🇫🇷 | TotalEnergies |
|
Energy | present | 0 | 13 | 12 | 1d ago |
| 4 | 🇨🇭 | Glencore |
|
Mining & Commodities | present | 0 | 29 | 28 | 2d ago |
| 5 | 🇬🇧 | BP |
|
Energy | present | 0 | 3 | 2 | 24d ago |
| 6 | 🇳🇱 | Stellantis |
|
Motor Vehicles | present | 0 | 1 | 0 | — |
| 7 | 🇩🇪 | Mercedes-Benz Group |
|
Motor Vehicles | present | 0 | 1 | 0 | — |
| 8 | 🇩🇪 | BMW Group |
|
Motor Vehicles | present | 0 | 1 | 0 | — |
| 9 | 🇪🇸 | Banco Santander |
|
Banking | present | 0 | 1 | 0 | — |
| 10 | 🇬🇧 | HSBC |
|
Banking | present | 0 | 1 | 0 | — |
| 11 | 🇩🇪 | Allianz |
|
Insurance | present | 0 | 1 | 0 | — |
| 12 | 🇫🇷 | AXA |
|
Insurance | present | 0 | 1 | 0 | — |
| 13 | 🇨🇭 | Nestle |
|
Food & Beverage | present | 0 | 3 | 2 | 1mo ago |
| 14 | 🇩🇪 | Siemens |
|
Industrials | present | 0 | 2 | 2 | 1mo ago |
| 15 | 🇫🇷 | Carrefour |
|
Retail | present | 0 | 1 | 0 | — |
| 16 | 🇩🇪 | Schwarz Group |
|
Retail | present | 0 | 1 | 0 | — |
| 17 | 🇫🇷 | EDF |
|
Utilities | present | 0 | 2 | 1 | 1mo ago |
| 18 | 🇫🇷 | Engie |
|
Utilities | present | 0 | 1 | 0 | — |
| 19 | 🇩🇪 | Deutsche Telekom |
|
Telecommunications | present | 0 | 2 | 1 | 2mo ago |
| 20 | 🇬🇧 | Vodafone |
|
Telecommunications | present | 0 | 2 | 1 | 3mo ago |
| 21 | 🇫🇷 | Orange |
|
Telecommunications | present | 0 | 9 | 8 | 3d ago |
| 22 | 🇪🇸 | Telefonica |
|
Telecommunications | present | 0 | 1 | 0 | — |
| 23 | 🇮🇹 | Enel |
|
Utilities | present | 0 | 1 | 0 | — |
| 24 | 🇮🇹 | Eni |
|
Energy | present | 0 | 1 | 0 | — |
| 25 | 🇳🇴 | Equinor |
|
Energy | absent | 0 | 1 | 1 | 4mo ago |
| 26 | 🇩🇰 | A.P. Moller-Maersk |
|
Shipping | present | 0 | 1 | 0 | — |
| 27 | 🇫🇷 | Airbus |
|
Aerospace & Defense | present | 🤖 1 | 1 | 0 | — |
| 28 | 🇩🇪 | SAP |
|
Technology | present | 0 | 11 | 10 | 17d ago |
| 29 | 🇳🇱 | ASML |
|
Technology | present | 0 | 1 | 0 | — |
| 30 | 🇫🇷 | LVMH |
|
Luxury | present | 0 | 1 | 0 | — |
| 31 | 🇫🇷 | L'Oreal |
|
Consumer Goods | present | 0 | 3 | 2 | 3mo ago |
| 32 | 🇫🇷 | Sanofi |
|
Pharmaceuticals | present | 0 | 1 | 0 | — |
| 33 | 🇩🇪 | Bayer |
|
Pharmaceuticals | present | 🤖 7 | 4 | 3 | 3d ago |
| 34 | 🇩🇪 | BASF |
|
Chemicals | present | 0 | 1 | 0 | — |
| 35 | 🇨🇭 | Novartis |
|
Pharmaceuticals | present | 0 | 1 | 0 | — |
| 36 | 🇨🇭 | Roche |
|
Pharmaceuticals | present | 0 | 1 | 0 | — |
| 37 | 🇬🇧 | AstraZeneca |
|
Pharmaceuticals | present | 0 | 2 | 1 | 2mo ago |
| 38 | 🇬🇧 | GSK |
|
Pharmaceuticals | present | 0 | 1 | 0 | — |
| 39 | 🇬🇧 | Unilever |
|
Consumer Goods | present | 0 | 1 | 0 | — |
| 40 | 🇬🇧 | Diageo |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 41 | 🇳🇱 | Heineken |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 42 | 🇧🇪 | AB InBev |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 43 | 🇪🇸 | Inditex |
|
Retail | redirect | 0 | 0 | 0 | — |
| 44 | 🇸🇪 | H&M Group |
|
Retail | present | 0 | 1 | 0 | — |
| 45 | 🇳🇱 | Ingka Group (IKEA) |
|
Retail | present | 0 | 1 | 0 | — |
| 46 | 🇩🇪 | Adidas |
|
Consumer Goods | present | 0 | 1 | 0 | — |
| 47 | 🇫🇮 | Nokia |
|
Technology | present | 0 | 1 | 0 | — |
| 48 | 🇸🇪 | Ericsson |
|
Technology | absent | 0 | 1 | 1 | 4mo ago |
| 49 | 🇳🇱 | Philips |
|
Health Technology | present | 0 | 1 | 0 | — |
| 50 | 🇫🇷 | Schneider Electric |
|
Industrials | present | 0 | 4 | 3 | 21d ago |
| 51 | 🇫🇷 | Saint-Gobain |
|
Industrials | present | 0 | 2 | 2 | 2mo ago |
| 52 | 🇫🇷 | Vinci |
|
Construction | present | 0 | 3 | 2 | 3mo ago |
| 53 | 🇫🇷 | Bouygues |
|
Construction | present | 0 | 1 | 0 | — |
| 54 | 🇱🇺 | ArcelorMittal |
|
Steel | present | 🤖 8 | 3 | 2 | 4mo ago |
| 55 | 🇩🇪 | Thyssenkrupp |
|
Industrials | present | 0 | 7 | 6 | 17d ago |
| 56 | 🇪🇸 | Repsol |
|
Energy | present | 0 | 1 | 0 | — |
| 57 | 🇦🇹 | OMV |
|
Energy | present | 0 | 1 | 0 | — |
| 58 | 🇵🇱 | Orlen |
|
Energy | present | 0 | 1 | 0 | — |
| 59 | 🇪🇸 | Iberdrola |
|
Utilities | present | 0 | 1 | 0 | — |
| 60 | 🇩🇪 | E.ON |
|
Utilities | present | 0 | 1 | 0 | — |
| 61 | 🇩🇪 | RWE |
|
Utilities | present | 0 | 1 | 0 | — |
| 62 | 🇫🇷 | Veolia |
|
Utilities | present | 0 | 2 | 1 | 2mo ago |
| 63 | 🇫🇷 | Danone |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 64 | 🇫🇷 | Pernod Ricard |
|
Food & Beverage | present | 0 | 2 | 1 | 2mo ago |
| 65 | 🇫🇷 | Kering |
|
Luxury | present | 0 | 1 | 0 | — |
| 66 | 🇫🇷 | Hermes |
|
Luxury | present | 0 | 4 | 3 | 1mo ago |
| 67 | 🇨🇭 | Richemont |
|
Luxury | present | 0 | 1 | 0 | — |
| 68 | 🇨🇭 | UBS |
|
Banking | present | 0 | 2 | 1 | 2mo ago |
| 69 | 🇩🇪 | Deutsche Bank |
|
Banking | present | 0 | 1 | 0 | — |
| 70 | 🇫🇷 | BNP Paribas |
|
Banking | present | 0 | 1 | 0 | — |
| 71 | 🇫🇷 | Credit Agricole |
|
Banking | present | 0 | 2 | 1 | 16d ago |
| 72 | 🇫🇷 | Societe Generale |
|
Banking | present | 0 | 2 | 2 | 1mo ago |
| 73 | 🇳🇱 | ING Group |
|
Banking | present | 0 | 1 | 0 | — |
| 74 | 🇬🇧 | Barclays |
|
Banking | present | 0 | 1 | 0 | — |
| 75 | 🇬🇧 | Lloyds Banking Group |
|
Banking | present | 0 | 2 | 1 | 1mo ago |
| 76 | 🇬🇧 | NatWest Group |
|
Banking | present | 0 | 1 | 0 | — |
| 77 | 🇮🇹 | Intesa Sanpaolo |
|
Banking | present | 0 | 1 | 0 | — |
| 79 | 🇫🇮 | Nordea |
|
Banking | present | 0 | 2 | 1 | 1mo ago |
| 80 | 🇨🇭 | Zurich Insurance |
|
Insurance | present | 0 | 1 | 0 | — |
| 81 | 🇩🇪 | Munich Re |
|
Insurance | present | 0 | 1 | 0 | — |
| 82 | 🇨🇭 | Swiss Re |
|
Insurance | present | 0 | 1 | 0 | — |
| 83 | 🇮🇹 | Generali |
|
Insurance | present | 0 | 1 | 0 | — |
| 84 | 🇬🇧 | Aviva |
|
Insurance | present | 0 | 1 | 0 | — |
| 85 | 🇬🇧 | Legal & General |
|
Insurance | present | 0 | 1 | 0 | — |
| 86 | 🇫🇷 | Renault Group |
|
Motor Vehicles | present | 0 | 4 | 4 | 16d ago |
| 87 | 🇸🇪 | Volvo Group |
|
Motor Vehicles | present | 0 | 1 | 0 | — |
| 88 | 🇸🇪 | Volvo Cars |
|
Motor Vehicles | present | 0 | 4 | 3 | 4mo ago |
| 89 | 🇮🇹 | Ferrari |
|
Motor Vehicles | present | 0 | 1 | 0 | — |
| 90 | 🇩🇪 | Porsche |
|
Motor Vehicles | present | 0 | 1 | 0 | — |
| 91 | 🇩🇪 | Continental |
|
Motor Vehicles | present | 0 | 2 | 1 | 3mo ago |
| 92 | 🇫🇷 | Michelin |
|
Motor Vehicles | present | 0 | 1 | 0 | — |
| 93 | 🇩🇪 | Bosch |
|
Industrials | present | 0 | 1 | 0 | — |
| 94 | 🇫🇷 | Safran |
|
Aerospace & Defense | present | 0 | 2 | 1 | 3mo ago |
| 95 | 🇫🇷 | Thales |
|
Aerospace & Defense | present | 🤖 1 | 2 | 1 | 4mo ago |
| 96 | 🇮🇹 | Leonardo |
|
Aerospace & Defense | present | 0 | 1 | 0 | — |
| 97 | 🇬🇧 | BAE Systems |
|
Aerospace & Defense | present | 0 | 2 | 1 | 4mo ago |
| 98 | 🇬🇧 | Rolls-Royce |
|
Aerospace & Defense | present | 0 | 1 | 0 | — |
| 99 | 🇩🇪 | Lufthansa Group |
|
Airlines | present | 🤖 9 | 1 | 0 | — |
| 100 | 🇫🇷 | Air France-KLM |
|
Airlines | present | 0 | 2 | 1 | 15d ago |
| 101 | 🇪🇸 | IAG |
|
Airlines | present | 0 | 1 | 0 | — |
| 102 | 🇮🇪 | Ryanair |
|
Airlines | present | 0 | 1 | 0 | — |
| 103 | 🇫🇷 | Accor |
|
Hospitality | present | 0 | 1 | 0 | — |
| 104 | 🇬🇧 | Compass Group |
|
Food Services | present | 0 | 1 | 0 | — |
| 105 | 🇫🇷 | Sodexo |
|
Food Services | present | 0 | 1 | 0 | — |
| 106 | 🇬🇧 | Tesco |
|
Retail | present | 0 | 1 | 0 | — |
| 108 | 🇳🇱 | Ahold Delhaize |
|
Retail | present | 0 | 2 | 1 | 3mo ago |
| 109 | 🇩🇪 | Metro AG |
|
Retail | present | 0 | 1 | 0 | — |
| 110 | 🇩🇪 | REWE Group |
|
Retail | present | 0 | 1 | 0 | — |
| 111 | 🇪🇸 | Mercadona |
|
Retail | present | 0 | 1 | 0 | — |
| 112 | 🇩🇪 | Zalando |
|
Retail | Not robots.txt | 0 | 0 | 0 | — |
| 113 | 🇸🇪 | Spotify |
|
Technology | present | 0 | 2 | 1 | 24d ago |
| 114 | 🇳🇱 | Adyen |
|
Technology | present | 0 | 1 | 0 | — |
| 115 | 🇳🇱 | Prosus |
|
Technology | present | 0 | 1 | 0 | — |
| 116 | 🇩🇪 | Infineon |
|
Technology | present | 0 | 4 | 3 | 11d ago |
| 117 | 🇨🇭 | STMicroelectronics |
|
Technology | present | 0 | 3 | 2 | 1d ago |
| 118 | 🇫🇷 | Capgemini |
|
Technology | present | 0 | 3 | 2 | 22d ago |
| 119 | 🇮🇪 | Accenture |
|
Technology | present | 0 | 1 | 0 | — |
| 120 | 🇬🇧 | RELX |
|
Information Services | present | 0 | 1 | 0 | — |
| 121 | 🇬🇧 | WPP |
|
Advertising | present | 🤖 2 | 1 | 0 | — |
| 122 | 🇫🇷 | Publicis Groupe |
|
Advertising | present | 0 | 1 | 0 | — |
| 123 | 🇩🇪 | Bertelsmann |
|
Media | present | 0 | 1 | 0 | — |
| 124 | 🇩🇪 | Axel Springer |
|
Publisher | present | 0 | 2 | 1 | 28d ago |
| 125 | 🇬🇧 | BT Group |
|
Telecommunications | present | 0 | 1 | 0 | — |
| 126 | 🇨🇭 | Swisscom |
|
Telecommunications | present | 0 | 1 | 0 | — |
| 127 | 🇳🇴 | Telenor |
|
Telecommunications | absent | 0 | 0 | 0 | — |
| 128 | 🇸🇪 | Telia |
|
Telecommunications | present | 0 | 1 | 0 | — |
| 129 | 🇳🇱 | KPN |
|
Telecommunications | present | 0 | 1 | 0 | — |
| 130 | 🇩🇪 | DHL Group |
|
Logistics | present | 0 | 3 | 2 | 3mo ago |
| 131 | 🇨🇭 | Kuehne+Nagel |
|
Logistics | present | 0 | 3 | 2 | 3mo ago |
| 132 | 🇩🇰 | DSV |
|
Logistics | present | 0 | 343 | 343 | 2h ago |
| 133 | 🇫🇷 | CMA CGM |
|
Shipping | present | 0 | 2 | 1 | 4mo ago |
| 134 | 🇩🇪 | Hapag-Lloyd |
|
Shipping | present | 0 | 1 | 0 | — |
| 135 | 🇪🇸 | Ferrovial |
|
Construction | present | 0 | 1 | 0 | — |
| 136 | 🇪🇸 | ACS Group |
|
Construction | present | 0 | 3 | 2 | 2d ago |
| 137 | 🇸🇪 | Skanska |
|
Construction | present | 0 | 1 | 0 | — |
| 138 | 🇨🇭 | Holcim |
|
Construction Materials | present | 0 | 1 | 0 | — |
| 139 | 🇩🇪 | Heidelberg Materials |
|
Construction Materials | present | 0 | 8 | 7 | 2mo ago |
| 140 | 🇮🇪 | CRH |
|
Construction Materials | present | 0 | 1 | 0 | — |
| 141 | 🇫🇷 | Air Liquide |
|
Chemicals | present | 0 | 1 | 0 | — |
| 142 | 🇳🇱 | AkzoNobel |
|
Chemicals | present | 0 | 1 | 0 | — |
| 143 | 🇧🇪 | Solvay |
|
Chemicals | present | 0 | 1 | 0 | — |
| 144 | 🇩🇪 | Evonik |
|
Chemicals | present | 0 | 2 | 2 | 3mo ago |
| 145 | 🇩🇪 | Henkel |
|
Consumer Goods | present | 0 | 1 | 0 | — |
| 146 | 🇩🇪 | Beiersdorf |
|
Consumer Goods | present | 0 | 1 | 0 | — |
| 147 | 🇬🇧 | Reckitt |
|
Consumer Goods | present | 0 | 1 | 0 | — |
| 148 | 🇬🇧 | Haleon |
|
Consumer Goods | present | 0 | 1 | 0 | — |
| 149 | 🇩🇰 | Novo Nordisk |
|
Pharmaceuticals | present | 0 | 1 | 0 | — |
| 150 | 🇩🇪 | Merck KGaA |
|
Pharmaceuticals | present | 0 | 1 | 0 | — |
| 151 | 🇩🇪 | Boehringer Ingelheim |
|
Pharmaceuticals | present | 0 | 2 | 1 | 1mo ago |
| 152 | 🇩🇪 | Fresenius |
|
Healthcare | present | 0 | 2 | 1 | 29d ago |
| 153 | 🇫🇷 | EssilorLuxottica |
|
Consumer Goods | present | 0 | 1 | 0 | — |
| 155 | 🇳🇱 | Booking.com |
|
Travel | present | 0 | 1 | 0 | — |
| 156 | 🇫🇷 | Decathlon |
|
Retail | present | 0 | 3 | 2 | 11d ago |
| 157 | 🇫🇷 | Leroy Merlin (Adeo) |
|
Retail | present | 0 | 1 | 0 | — |
| 158 | 🇪🇸 | El Corte Ingles |
|
Retail | present | 🤖 2 | 3 | 3 | 25d ago |
| 159 | 🇪🇸 | Naturgy |
|
Utilities | present | 0 | 1 | 0 | — |
| 160 | 🇸🇪 | Vattenfall |
|
Utilities | present | 0 | 1 | 0 | — |