AliExpress / robots.txt
← back to aliexpress.com · change detected 22d ago (2026-10-15T12:33:19Z) · modified +22 −3
What changed
- 🤖 AIBlocked Amazonbot (Amazon) entirely
- 🤖 AIBlocked Bytespider (ByteDance) entirely
- 🤖 AIBlocked CCBot (Common Crawl) entirely
- 🤖 AIBlocked ClaudeBot (Anthropic) entirely
- 🤖 AIBlocked Google-Extended (Google) entirely
- 🤖 AIBlocked GPTBot (OpenAI) entirely
- Blocked user-agent ‘SemrushBot’ entirely
- Removed all rules for user-agent ‘Googlebot’
- Added Disallow: */aeglodetailweb/api/msite/item?productId* for *
- Added Disallow: */aeglodetailweb/api/msite/item?productId* for YandexBot
Diff old (2026-09-24T10:55:22Z) → new (2026-10-15T12:33:19Z)
| @@ -1,6 +1,3 @@ | ||
| 1 | - User-agent: Googlebot | |
| 2 | - Disallow: */aeglodetailweb/api/msite/item?productId* | |
| 3 | - | |
| 4 | 1 | User-agent: * |
| 5 | 2 | Disallow: /items/* |
| 6 | 3 | Disallow: /bin/* |
| @@ -40,6 +37,7 @@ | ||
| 40 | 37 | Disallow: /i/api/* |
| 41 | 38 | Disallow: /gcp/300000414/regulatedinformation |
| 42 | 39 | Disallow: /img/wsproduct/sku/* |
| 40 | + Disallow: */aeglodetailweb/api/msite/item?productId* | |
| 43 | 41 | |
| 44 | 42 | User-agent: YandexBot |
| 45 | 43 | Crawl-delay: 0.5 |
| @@ -77,7 +75,28 @@ | ||
| 77 | 75 | Disallow: /i/api/* |
| 78 | 76 | Disallow: /gcp/300000414/regulatedinformation |
| 79 | 77 | Disallow: /img/wsproduct/sku/* |
| 78 | + Disallow: */aeglodetailweb/api/msite/item?productId* | |
| 80 | 79 | |
| 80 | + User-agent: GPTBot | |
| 81 | + Disallow: / | |
| 82 | + | |
| 83 | + User-agent: CCBot | |
| 84 | + Disallow: / | |
| 85 | + | |
| 86 | + User-agent: Google-Extended | |
| 87 | + Disallow: / | |
| 88 | + | |
| 89 | + User-agent: ClaudeBot | |
| 90 | + Disallow: / | |
| 91 | + | |
| 92 | + User-agent: Amazonbot | |
| 93 | + Disallow: / | |
| 94 | + | |
| 95 | + User-agent: Bytespider | |
| 96 | + Disallow: / | |
| 97 | + | |
| 98 | + User-agent: SemrushBot | |
| 99 | + Disallow: / | |
| 81 | 100 | |
| 82 | 101 | Sitemap: https://www.aliexpress.com/sitemap/en-https-item-item-2026-P0-sitemap-index.xml |
| 83 | 102 | Sitemap: https://www.aliexpress.com/sitemap/en-https-item-item-2026-P1-sitemap-index.xml |