E-commerce
The archive — every monitored site, its robots.txt history and AI-crawler stance.
All sites
Top 1000 websites
US 500
Nasdaq 100
AI models
Cybersecurity
SEO & AI search
Creative rights-holders
Knowledge & UGC
Academic & open research
Universities
National indices
Banks & fintech
E-commerce
European companies
Social Networks
Governments
Publishers
| # | Company | Domain | Sector | robots.txt | AI bots blocked | Snapshots | Changes | Last change | |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 🇺🇸 | Amazon |
amazon.com
|
Retailing | present | 🤖 24 | 2 | 1 | 1mo ago |
| 2 | 🇺🇸 | Walmart |
|
Retailing | present | 0 | 3 | 2 | 28d ago |
| 3 | 🇨🇳 | Alibaba Group |
|
E-commerce | present | 🤖 2 | 1 | 0 | — |
| 4 | 🇨🇳 | JD.com |
|
E-commerce | present | 0 | 1 | 0 | — |
| 5 | 🇺🇸 | Temu |
|
E-commerce | present | 0 | 1 | 0 | — |
| 6 | 🇨🇳 | Shein |
|
E-commerce | present | 0 | 1 | 0 | — |
| 7 | 🇺🇸 | eBay |
|
Technology | present | 🤖 9 | 5 | 4 | 2mo ago |
| 8 | 🇯🇵 | Rakuten |
|
E-commerce | present | 0 | 1 | 0 | — |
| 9 | 🇦🇷 | Mercado Libre |
|
E-commerce | present | 🤖 7 | 1 | 0 | — |
| 10 | 🇺🇸 | Target |
|
Retailing | present | 0 | 3 | 3 | 21d ago |
| 11 | 🇺🇸 | Costco Wholesale |
|
Retailing | present | 0 | 1 | 0 | — |
| 12 | 🇺🇸 | Wayfair |
|
Retailing | present | 0 | 1 | 0 | — |
| 13 | 🇺🇸 | Chewy |
|
Retailing | present | 🤖 1 | 2 | 1 | 12d ago |
| 14 | 🇺🇸 | Best Buy |
|
Retailing | present | 0 | 5 | 4 | 4d ago |
| 15 | 🇺🇸 | Etsy |
|
E-commerce | present | 0 | 62 | 61 | 2d ago |
| 16 | 🇺🇸 | Home Depot |
|
Retailing | present | 0 | 2 | 1 | 24d ago |
| 17 | 🇺🇸 | Adorama |
|
Retailer | present | 0 | 8 | 7 | 13d ago |
| 18 | 🇨🇦 | Shopify |
shopify.com
|
E-commerce | present | 0 | 1 | 0 | — |
| 19 | 🇨🇳 | AliExpress |
|
E-commerce | present | 🤖 6 | 4 | 3 | 25d ago |
| 20 | 🇺🇸 | Wish |
|
E-commerce | present | 0 | 1 | 0 | — |