National indices
The archive — every monitored site, its robots.txt history and AI-crawler stance.
All sites
Top 1000 websites
US 500
Nasdaq 100
AI labs
Cybersecurity
SEO & AI search
Creative rights-holders
Knowledge & UGC
National indices
Banks & fintech
European companies
Social Networks
Governments
Publishers
| # | Company | Domain | Sector | robots.txt | AI bots blocked | Snapshots | Changes | Last change | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 🇬🇧 | AstraZeneca |
|
Pharmaceuticals | present | 0 | 1 | 0 | — |
| 2 | 🇬🇧 | Shell |
|
Energy | present | 0 | 1 | 0 | — |
| 3 | 🇬🇧 | HSBC |
|
Banking | present | 0 | 1 | 0 | — |
| 4 | 🇬🇧 | Unilever |
|
Consumer Goods | present | 0 | 1 | 0 | — |
| 5 | 🇬🇧 | BP |
|
Energy | present | 0 | 1 | 0 | — |
| 6 | 🇬🇧 | GSK |
|
Pharmaceuticals | present | 0 | 1 | 0 | — |
| 7 | 🇬🇧 | Rio Tinto |
|
Materials | present | 0 | 1 | 0 | — |
| 8 | 🇬🇧 | Diageo |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 9 | 🇬🇧 | British American Tobacco |
|
Consumer Goods | present | 0 | 1 | 0 | — |
| 10 | 🇬🇧 | Barclays |
|
Banking | present | 0 | 1 | 0 | — |
| 11 | 🇩🇪 | SAP |
|
Technology | present | 0 | 1 | 0 | — |
| 12 | 🇩🇪 | Siemens |
|
Industrials | present | 0 | 1 | 0 | — |
| 13 | 🇩🇪 | Allianz |
|
Insurance | present | 0 | 1 | 0 | — |
| 14 | 🇩🇪 | Volkswagen Group |
|
Motor Vehicles | present | 0 | 1 | 0 | — |
| 15 | 🇩🇪 | Mercedes-Benz Group |
|
Motor Vehicles | present | 0 | 1 | 0 | — |
| 16 | 🇩🇪 | BMW |
|
Motor Vehicles | present | 0 | 1 | 0 | — |
| 17 | 🇩🇪 | Deutsche Telekom |
|
Telecommunications | present | 0 | 1 | 0 | — |
| 18 | 🇩🇪 | Bayer |
|
Pharmaceuticals | present | 0 | 1 | 0 | — |
| 19 | 🇩🇪 | BASF |
|
Chemicals | present | 0 | 1 | 0 | — |
| 20 | 🇩🇪 | Adidas |
|
Fashion and Apparel | present | 0 | 2 | 1 | 2h ago |
| 21 | 🇫🇷 | LVMH |
|
Luxury | present | 0 | 1 | 0 | — |
| 22 | 🇫🇷 | TotalEnergies |
|
Energy | present | 0 | 1 | 0 | — |
| 23 | 🇫🇷 | L'Oreal |
|
Consumer Goods | present | 0 | 1 | 0 | — |
| 24 | 🇫🇷 | Sanofi |
|
Pharmaceuticals | present | 0 | 1 | 0 | — |
| 25 | 🇫🇷 | Airbus |
|
Aerospace & Defense | present | 🤖 1 | 1 | 0 | — |
| 26 | 🇫🇷 | Hermes |
|
Luxury | present | 0 | 1 | 0 | — |
| 27 | 🇫🇷 | Schneider Electric |
|
Industrials | present | 0 | 1 | 0 | — |
| 28 | 🇫🇷 | BNP Paribas |
|
Banking | present | 0 | 1 | 0 | — |
| 29 | 🇫🇷 | AXA |
|
Insurance | present | 0 | 1 | 0 | — |
| 30 | 🇫🇷 | Danone |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 31 | 🇯🇵 | Toyota |
|
Motor Vehicles | present | 🤖 13 | 1 | 0 | — |
| 32 | 🇯🇵 | Sony |
|
Consumer Goods | present | 0 | 1 | 0 | — |
| 33 | 🇯🇵 | Nintendo |
|
Video Games Consoles and Accessories | present | 🤖 15 | 1 | 0 | — |
| 34 | 🇯🇵 | SoftBank Group |
|
Telecommunications | present | 0 | 1 | 0 | — |
| 35 | 🇯🇵 | Honda |
|
Motor Vehicles | absent | 0 | 0 | 0 | — |
| 36 | 🇯🇵 | Hitachi |
|
Industrials | present | 0 | 1 | 0 | — |
| 37 | 🇯🇵 | Keyence |
|
Industrials | present | 0 | 1 | 0 | — |
| 38 | 🇯🇵 | Fast Retailing |
|
Retailing | absent | 0 | 0 | 0 | — |
| 39 | 🇯🇵 | Nippon Telegraph and Telephone |
|
Telecommunications | absent | 0 | 0 | 0 | — |
| 40 | 🇯🇵 | Mitsubishi UFJ Financial Group |
|
Banking | absent | 0 | 0 | 0 | — |
| 41 | 🇨🇦 | Royal Bank of Canada |
|
Banking | present | 0 | 1 | 0 | — |
| 42 | 🇨🇦 | Shopify |
shopify.com
|
E-commerce and Shopping | present | 0 | 1 | 0 | — |
| 43 | 🇨🇦 | Brookfield |
|
Finance | present | 0 | 1 | 0 | — |
| 44 | 🇨🇦 | Enbridge |
|
Energy | present | 0 | 1 | 0 | — |
| 45 | 🇨🇦 | Canadian National Railway |
|
Transportation | present | 0 | 1 | 0 | — |
| 46 | 🇨🇦 | Toronto-Dominion Bank |
|
Banking | present | 0 | 1 | 0 | — |
| 47 | 🇦🇺 | BHP |
|
Materials | blocked | 0 | 0 | 0 | — |
| 48 | 🇦🇺 | Commonwealth Bank |
|
Banking | present | 0 | 1 | 0 | — |
| 49 | 🇦🇺 | CSL |
|
Pharmaceuticals | present | 0 | 1 | 0 | — |
| 50 | 🇦🇺 | Telstra |
|
Telecommunications | present | 🤖 5 | 1 | 0 | — |
| 51 | 🇦🇺 | Westpac |
|
Banking | present | 0 | 1 | 0 | — |
| 52 | 🇮🇳 | Reliance Industries |
|
Industrials | wrong response | 0 | 0 | 0 | — |
| 53 | 🇮🇳 | Tata Consultancy Services |
|
Technology | present | 0 | 1 | 0 | — |
| 54 | 🇮🇳 | Infosys |
|
Technology | present | 0 | 1 | 0 | — |
| 55 | 🇮🇳 | HDFC Bank |
|
Banking | blocked | 0 | 0 | 0 | — |
| 56 | 🇮🇳 | ICICI Bank |
|
Banking | present | 0 | 1 | 0 | — |
| 57 | 🇮🇳 | Bharti Airtel |
|
Telecommunications | present | 0 | 1 | 0 | — |
| 58 | 🇨🇭 | Nestle |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 59 | 🇨🇭 | Novartis |
|
Pharmaceuticals | present | 0 | 1 | 0 | — |
| 60 | 🇨🇭 | Roche |
|
Pharmaceuticals | present | 0 | 1 | 0 | — |
| 61 | 🇨🇭 | UBS |
|
Banking | present | 0 | 1 | 0 | — |
| 62 | 🇰🇷 | Samsung Electronics |
|
Technology | present | 0 | 1 | 0 | — |
| 63 | 🇰🇷 | SK Hynix |
|
Semiconductors | present | 0 | 1 | 0 | — |
| 64 | 🇰🇷 | Hyundai Motor |
|
Motor Vehicles | present | 0 | 1 | 0 | — |
| 65 | 🇰🇷 | NAVER |
|
Technology | present | 0 | 1 | 0 | — |
| 66 | 🇰🇷 | LG Corp |
|
Conglomerate | present | 0 | 1 | 0 | — |
| 67 | 🇨🇳 | Alibaba Group |
|
E-commerce | present | 🤖 2 | 1 | 0 | — |
| 68 | 🇨🇳 | Tencent |
|
Technology | present | 0 | 1 | 0 | — |
| 69 | 🇨🇳 | BYD |
|
Motor Vehicles | absent | 0 | 0 | 0 | — |
| 70 | 🇨🇳 | Xiaomi |
|
Technology | present | 0 | 1 | 0 | — |
| 71 | 🇨🇳 | Meituan |
|
E-commerce | absent | 0 | 0 | 0 | — |
| 72 | 🇨🇳 | Baidu |
|
Technology | present | 0 | 1 | 0 | — |
| 73 | 🇹🇼 | TSMC |
|
Semiconductors | present | 0 | 1 | 0 | — |
| 74 | 🇹🇼 | Hon Hai (Foxconn) |
|
Industrials | present | 0 | 1 | 0 | — |
| 75 | 🇹🇼 | MediaTek |
|
Semiconductors | present | 0 | 1 | 0 | — |
| 76 | 🇸🇬 | DBS Group |
|
Banking | present | 0 | 1 | 0 | — |
| 77 | 🇸🇬 | Grab |
|
Technology | present | 0 | 1 | 0 | — |
| 78 | 🇸🇬 | Sea Limited |
|
E-commerce | present | 0 | 1 | 0 | — |
| 79 | 🇧🇷 | Petrobras |
|
Energy | present | 0 | 1 | 0 | — |
| 80 | 🇧🇷 | Vale |
|
Materials | present | 0 | 1 | 0 | — |
| 81 | 🇧🇷 | Itaú Unibanco |
|
Banking | present | 0 | 1 | 0 | — |
| 82 | 🇧🇷 | Ambev |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 83 | 🇧🇷 | JBS |
|
Food & Beverage | blocked | 0 | 0 | 0 | — |
| 84 | 🇲🇽 | FEMSA |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 85 | 🇲🇽 | América Móvil |
|
Telecommunications | unreachable | 0 | 0 | 0 | — |
| 86 | 🇲🇽 | Grupo Bimbo |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 87 | 🇲🇽 | Cemex |
|
Materials | present | 0 | 1 | 0 | — |
| 88 | 🇿🇦 | Naspers |
|
Technology | present | 0 | 1 | 0 | — |
| 89 | 🇿🇦 | Anglo American |
|
Materials | present | 0 | 1 | 0 | — |
| 90 | 🇿🇦 | MTN Group |
|
Telecommunications | present | 0 | 1 | 0 | — |
| 91 | 🇸🇦 | Saudi Aramco |
|
Energy | present | 0 | 1 | 0 | — |
| 92 | 🇸🇦 | SABIC |
|
Chemicals | blocked | 0 | 0 | 0 | — |
| 93 | 🇦🇪 | Emirates NBD |
|
Banking | blocked | 0 | 0 | 0 | — |
| 94 | 🇸🇪 | Ericsson |
|
Technology | present | 0 | 1 | 0 | — |
| 95 | 🇸🇪 | Volvo Group |
|
Motor Vehicles | present | 0 | 1 | 0 | — |
| 96 | 🇸🇪 | H&M |
|
Retailing | present | 0 | 1 | 0 | — |
| 97 | 🇸🇪 | Spotify |
|
Technology | present | 0 | 1 | 0 | — |
| 98 | 🇪🇸 | Inditex |
|
Retail | forced redirect | 0 | 0 | 0 | — |
| 99 | 🇪🇸 | Iberdrola |
|
Utilities | present | 0 | 1 | 0 | — |
| 100 | 🇪🇸 | Banco Santander |
|
Banking | present | 0 | 1 | 0 | — |
| 101 | 🇪🇸 | Telefonica |
|
Telecommunications | present | 0 | 1 | 0 | — |
| 102 | 🇮🇹 | Eni |
|
Energy | present | 0 | 1 | 0 | — |
| 103 | 🇮🇹 | Enel |
|
Utilities | present | 0 | 1 | 0 | — |
| 104 | 🇮🇹 | Ferrari |
|
Motor Vehicles | present | 0 | 1 | 0 | — |
| 105 | 🇮🇹 | Intesa Sanpaolo |
|
Banking | present | 0 | 1 | 0 | — |
| 106 | 🇳🇱 | ASML |
|
Technology | present | 0 | 1 | 0 | — |
| 107 | 🇳🇱 | ING Group |
|
Banking | present | 0 | 1 | 0 | — |
| 108 | 🇳🇱 | Heineken |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 109 | 🇳🇴 | Equinor |
|
Energy | absent | 0 | 0 | 0 | — |
| 110 | 🇳🇴 | DNB |
|
Banking | present | 0 | 1 | 0 | — |
| 111 | 🇩🇰 | Novo Nordisk |
|
Pharmaceuticals | present | 0 | 1 | 0 | — |
| 112 | 🇩🇰 | A.P. Moller-Maersk |
|
Shipping | present | 0 | 1 | 0 | — |
| 113 | 🇫🇮 | Nokia |
|
Technology | present | 0 | 1 | 0 | — |
| 114 | 🇵🇱 | PKO Bank Polski |
|
Banking | present | 0 | 1 | 0 | — |
| 115 | 🇹🇷 | Koç Holding |
|
Conglomerate | present | 0 | 1 | 0 | — |
| 116 | 🇮🇩 | Bank Central Asia |
|
Banking | present | 0 | 3 | 2 | 2h ago |
| 117 | 🇮🇩 | Telkom Indonesia |
|
Telecommunications | present | 0 | 1 | 0 | — |
| 118 | 🇹🇭 | PTT |
|
Energy | present | 0 | 1 | 0 | — |
| 119 | 🇲🇾 | Petronas |
|
Energy | present | 0 | 1 | 0 | — |
| 120 | 🇲🇾 | Maybank |
|
Banking | unreachable | 0 | 0 | 0 | — |
| 121 | 🇵🇭 | SM Investments |
|
Conglomerate | blocked | 0 | 0 | 0 | — |
| 122 | 🇦🇷 | Mercado Libre |
|
E-commerce | present | 🤖 7 | 1 | 0 | — |
| 123 | 🇦🇷 | YPF |
|
Energy | present | 0 | 1 | 0 | — |
| 124 | 🇨🇱 | Falabella |
|
Retailing | blocked | 0 | 0 | 0 | — |
| 125 | 🇨🇴 | Ecopetrol |
|
Energy | present | 0 | 1 | 0 | — |
| 126 | 🇮🇱 | Teva Pharmaceutical |
|
Pharmaceuticals | present | 0 | 1 | 0 | — |
| 127 | 🇮🇱 | Check Point |
|
Technology | present | 0 | 1 | 0 | — |
| 128 | 🇦🇹 | OMV |
|
Energy | present | 0 | 1 | 0 | — |
| 129 | 🇵🇹 | Galp |
|
Energy | present | 0 | 1 | 0 | — |
| 130 | 🇳🇿 | Fonterra |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 131 | 🇬🇧 | Arm Holdings |
|
Semiconductors | present | 0 | 1 | 0 | — |
| 132 | 🇯🇵 | KIOXIA |
|
Technology | present | 0 | 1 | 0 | — |
| 133 | 🇨🇳 | CATL |
|
Industrials | forced redirect | 0 | 0 | 0 | — |
| 134 | 🇨🇳 | Zhongji Innolight |
|
Technology | present | 0 | 1 | 0 | — |
| 135 | 🇨🇳 | Kweichow Moutai |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 136 | 🇨🇳 | China Mobile |
|
Telecommunications | absent | 0 | 0 | 0 | — |
| 137 | 🇨🇳 | PetroChina |
|
Energy | forced redirect | 0 | 0 | 0 | — |
| 138 | 🇯🇵 | Tokyo Electron |
|
Technology | absent | 0 | 0 | 0 | — |
| 139 | 🇦🇪 | International Holding Company |
|
Conglomerate | unreachable | 0 | 0 | 0 | — |