Nasdaq 100
The archive — every monitored site, its robots.txt history and AI-crawler stance.
All sites
Top 1000 websites
US 500
Nasdaq 100
AI labs
Cybersecurity
SEO & AI search
Creative rights-holders
Knowledge & UGC
National indices
Banks & fintech
European companies
Social Networks
Governments
Publishers
| # | Company | Domain | Sector | robots.txt | AI bots blocked | Snapshots | Changes | Last change | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 🇺🇸 | Apple |
|
Technology | present | 0 | 1 | 0 | — |
| 2 | 🇺🇸 | Microsoft |
|
Technology | present | 0 | 1 | 0 | — |
| 3 | 🇺🇸 | Nvidia |
|
Technology | present | 0 | 1 | 0 | — |
| 4 | 🇺🇸 | Amazon |
amazon.com
|
Retailing | present | 🤖 23 | 1 | 0 | — |
| 5 | 🇺🇸 | Alphabet |
|
Technology | present | 0 | 1 | 0 | — |
| 6 | 🇺🇸 | Meta Platforms |
|
Technology | present | 0 | 1 | 0 | — |
| 7 | 🇺🇸 | Broadcom |
|
Technology | present | 0 | 1 | 0 | — |
| 8 | 🇺🇸 | Tesla |
|
Motor Vehicles | present | 0 | 1 | 0 | — |
| 9 | 🇺🇸 | Costco Wholesale |
|
Retailing | present | 0 | 1 | 0 | — |
| 10 | 🇺🇸 | Netflix |
|
Media | present | 0 | 1 | 0 | — |
| 11 | 🇺🇸 | Advanced Micro Devices |
|
Technology | present | 0 | 1 | 0 | — |
| 12 | 🇺🇸 | PepsiCo |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 13 | 🇺🇸 | Adobe |
|
Technology | present | 0 | 1 | 0 | — |
| 14 | 🇺🇸 | Linde |
|
Chemicals | present | 0 | 1 | 0 | — |
| 15 | 🇺🇸 | Cisco Systems |
|
Technology | present | 0 | 1 | 0 | — |
| 16 | 🇺🇸 | Qualcomm |
|
Technology | present | 0 | 1 | 0 | — |
| 17 | 🇺🇸 | T-Mobile US |
|
Telecommunications | blocked | 0 | 0 | 0 | — |
| 18 | 🇺🇸 | Comcast |
|
Media | present | 0 | 1 | 0 | — |
| 19 | 🇺🇸 | Intuit |
|
Technology | present | 0 | 1 | 0 | — |
| 20 | 🇺🇸 | Texas Instruments |
|
Technology | present | 0 | 1 | 0 | — |
| 21 | 🇺🇸 | Amgen |
|
Health Care | present | 0 | 3 | 2 | 5h ago |
| 22 | 🇺🇸 | Honeywell International |
|
Industrials | present | 0 | 1 | 0 | — |
| 23 | 🇺🇸 | Intel |
|
Technology | present | 0 | 1 | 0 | — |
| 24 | 🇺🇸 | Starbucks |
|
Food Services | present | 0 | 1 | 0 | — |
| 25 | 🇺🇸 | Applied Materials |
|
Technology | present | 0 | 1 | 0 | — |
| 26 | 🇺🇸 | Booking Holdings |
|
Technology | present | 0 | 1 | 0 | — |
| 27 | 🇺🇸 | Mondelez International |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 28 | 🇺🇸 | ADP |
|
Business Services | present | 0 | 1 | 0 | — |
| 29 | 🇺🇸 | Gilead Sciences |
|
Health Care | present | 0 | 1 | 0 | — |
| 30 | 🇺🇸 | Vertex Pharmaceuticals |
|
Health Care | present | 0 | 1 | 0 | — |
| 31 | 🇺🇸 | Lam Research |
|
Technology | present | 0 | 1 | 0 | — |
| 32 | 🇺🇸 | Analog Devices |
|
Technology | present | 0 | 1 | 0 | — |
| 33 | 🇺🇸 | KLA |
|
Technology | present | 0 | 1 | 0 | — |
| 34 | 🇺🇸 | Regeneron Pharmaceuticals |
|
Health Care | present | 0 | 1 | 0 | — |
| 35 | 🇺🇸 | Palo Alto Networks |
|
Technology | present | 0 | 1 | 0 | — |
| 36 | 🇺🇸 | Snowflake |
|
Technology | present | 0 | 1 | 0 | — |
| 37 | 🇺🇸 | Micron Technology |
|
Technology | present | 0 | 1 | 0 | — |
| 38 | 🇺🇸 | Intuitive Surgical |
|
Health Care | present | 0 | 1 | 0 | — |
| 39 | 🇺🇸 | CrowdStrike |
|
Computer Security | present | 0 | 1 | 0 | — |
| 40 | 🇺🇸 | Airbnb |
|
Technology | present | 0 | 1 | 0 | — |
| 41 | 🇺🇸 | Cadence Design Systems |
|
Technology | blocked | 0 | 0 | 0 | — |
| 42 | 🇺🇸 | Synopsys |
|
Technology | present | 0 | 1 | 0 | — |
| 43 | 🇺🇸 | Marriott International |
|
Hotels & Leisure | present | 0 | 1 | 0 | — |
| 44 | 🇺🇸 | Workday |
|
Technology | present | 0 | 1 | 0 | — |
| 45 | 🇺🇸 | Fortinet |
|
Computer Security | present | 🤖 1 | 1 | 0 | — |
| 46 | 🇺🇸 | Paychex |
|
Business Services | blocked | 0 | 0 | 0 | — |
| 47 | 🇺🇸 | O'Reilly Automotive |
|
Retailing | present | 0 | 1 | 0 | — |
| 48 | 🇺🇸 | Charter Communications |
|
Telecommunications | present | 0 | 1 | 0 | — |
| 49 | 🇺🇸 | Keurig Dr Pepper |
|
Food & Beverage | present | 0 | 1 | 0 | — |
| 50 | 🇺🇸 | Marvell Technology |
|
Semiconductors | present | 0 | 1 | 0 | — |
| 51 | 🇺🇸 | PayPal Holdings |
|
Financials | present | 0 | 1 | 0 | — |
| 52 | 🇺🇸 | American Electric Power |
|
Energy | absent | 0 | 0 | 0 | — |
| 53 | 🇺🇸 | Exelon |
|
Energy | present | 0 | 1 | 0 | — |
| 54 | 🇺🇸 | Copart |
|
Business Services | present | 0 | 1 | 0 | — |
| 55 | 🇺🇸 | Ross Stores |
|
Retailing | present | 0 | 1 | 0 | — |
| 56 | 🇺🇸 | Cintas |
|
Industrials | present | 0 | 1 | 0 | — |
| 57 | 🇺🇸 | Fastenal |
|
Wholesale | present | 0 | 1 | 0 | — |
| 58 | 🇺🇸 | Diamondback Energy |
|
Energy | present | 🤖 1 | 1 | 0 | — |
| 59 | 🇺🇸 | Datadog |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 60 | 🇺🇸 | Autodesk |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 61 | 🇺🇸 | Microchip Technology |
|
Semiconductors | present | 0 | 1 | 0 | — |
| 62 | 🇺🇸 | IDEXX Laboratories |
|
Health Care | present | 0 | 1 | 0 | — |
| 63 | 🇺🇸 | CSX |
|
Transportation | blocked | 0 | 0 | 0 | — |
| 64 | 🇺🇸 | Baker Hughes |
|
Energy | wrong response | 0 | 0 | 0 | — |
| 65 | 🇺🇸 | Old Dominion Freight Line |
|
Transportation | present | 0 | 1 | 0 | — |
| 66 | 🇺🇸 | Biogen |
|
Health Care | present | 0 | 1 | 0 | — |
| 67 | 🇺🇸 | GE HealthCare |
|
Health Care | present | 0 | 1 | 0 | — |
| 68 | 🇺🇸 | Constellation Energy |
|
Energy | present | 0 | 1 | 0 | — |
| 69 | 🇺🇸 | Lululemon Athletica |
|
Apparel | present | 🤖 1 | 1 | 0 | — |
| 70 | 🇺🇸 | Electronic Arts |
|
Technology | present | 🤖 4 | 1 | 0 | — |
| 71 | 🇺🇸 | DexCom |
|
Health Care | present | 0 | 1 | 0 | — |
| 72 | 🇺🇸 | Kraft Heinz |
|
Food & Beverage | absent | 0 | 0 | 0 | — |
| 73 | 🇺🇸 | Zscaler |
|
Computer Security | present | 0 | 1 | 0 | — |
| 74 | 🇺🇸 | Onsemi |
|
Technology | present | 0 | 1 | 0 | — |
| 75 | 🇺🇸 | CDW |
|
Technology | unreachable | 0 | 0 | 0 | — |
| 76 | 🇺🇸 | Ansys |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 77 | 🇺🇸 | The Trade Desk |
|
Business Services | present | 0 | 1 | 0 | — |
| 78 | 🇺🇸 | Cognizant |
|
Technology | present | 0 | 1 | 0 | — |
| 79 | 🇺🇸 | Atlassian |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 80 | 🇺🇸 | Xcel Energy |
|
Energy | absent | 0 | 0 | 0 | — |
| 81 | 🇺🇸 | Paccar |
|
Motor Vehicles | unreachable | 0 | 0 | 0 | — |
| 82 | 🇺🇸 | Verisign |
|
Technology | present | 0 | 1 | 0 | — |
| 83 | 🇺🇸 | Warner Bros. Discovery |
|
Media | present | 0 | 1 | 0 | — |
| 84 | 🇺🇸 | GlobalFoundries |
|
Semiconductors | present | 0 | 1 | 0 | — |
| 85 | 🇺🇸 | Align Technology |
|
Health Care | present | 0 | 1 | 0 | — |
| 86 | 🇺🇸 | Take-Two Interactive |
|
Technology | present | 0 | 1 | 0 | — |
| 87 | 🇺🇸 | Dollar Tree |
|
Retailing | present | 0 | 1 | 0 | — |
| 88 | 🇺🇸 | Sirius XM |
|
Media | present | 0 | 1 | 0 | — |
| 89 | 🇺🇸 | Walgreens Boots Alliance |
|
Health Care | present | 0 | 1 | 0 | — |
| 90 | 🇺🇸 | Enphase Energy |
|
Energy | present | 0 | 1 | 0 | — |
| 91 | 🇺🇸 | First Solar |
|
Energy | present | 0 | 1 | 0 | — |
| 92 | 🇺🇸 | Super Micro Computer |
|
Technology | present | 0 | 1 | 0 | — |
| 93 | 🇺🇸 | MongoDB |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 94 | 🇺🇸 | Okta |
|
Computer Security | present | 0 | 1 | 0 | — |
| 95 | 🇺🇸 | DoorDash |
|
Technology | present | 0 | 1 | 0 | — |
| 96 | 🇺🇸 | Coinbase |
|
Finance | present | 0 | 1 | 0 | — |
| 97 | 🇺🇸 | Expedia Group |
|
Travel and Tourism | present | 0 | 1 | 0 | — |
| 98 | 🇺🇸 | eBay |
|
Technology | present | 🤖 9 | 1 | 0 | — |
| 99 | 🇺🇸 | Zoom Communications |
|
Programming and Developer Software | present | 0 | 1 | 0 | — |
| 100 | 🇺🇸 | Roper Technologies |
|
Programming and Developer Software | present | 🤖 1 | 1 | 0 | — |
| 101 | 🇬🇧 | Arm Holdings |
|
Semiconductors | present | 0 | 1 | 0 | — |
| 102 | 🇺🇸 | Palantir |
|
Technology | present | 0 | 1 | 0 | — |