NeuralCrawl

🇨🇳 AliExpress

aliexpress.com · E-commerce · rank #19 · E-commerce · live robots.txt ↗

AI crawler access (latest snapshot, 2h ago)

⛔blocked restricted ✅allowed faded = inherited from the * wildcard group

✅GPTBot
✅ChatGPT-User
✅OAI-SearchBot
✅ClaudeBot
✅Claude-User
✅Claude-SearchBot
✅anthropic-ai
✅Claude-Web
✅CCBot
✅Google-Extended
✅Applebot-Extended
✅PerplexityBot
✅Perplexity-User
✅Bytespider
✅Amazonbot
✅FacebookBot
✅meta-externalagent
✅meta-externalfetcher
✅cohere-ai
✅AI2Bot
✅Diffbot
✅omgili
✅YouBot
✅DuckAssistBot
✅MistralAI-User
✅PanguBot
✅Timpibot

Current robots.txt 2005 bytes · sha256 584457a258b4 · raw 

User-agent: Googlebot
Disallow: */aeglodetailweb/api/msite/item?productId*

User-agent: AdsBot-Google
Disallow: /i/*

User-agent: AdsBot-Google-Mobile
Disallow: /i/*

User-agent: *
Disallow: /items/*
Disallow: /bin/*
Disallow: /search/*
Allow: /wholesale.html$
Allow: /wholesale-page-*.html
Disallow: /wholesale*
Disallow: /productdetail/*
Allow: /api/data_homepage.do
Disallow: /api/*
Disallow: /api*.do
Disallow: /apps/*
Disallow: /downloads/*
Disallow: /wishlist/*
Disallow: /shopcart/*
Disallow: /brands/*
Disallow: /cp/*
Disallow: /item-img/*
Disallow: /product/*
Disallow: /p4p*list.html
Disallow: /orderList.html
Disallow: /ws*api-server.html$
Disallow: /*mmend.htm$
Disallow: /*Ajax.htm$
Disallow: /*ajax.htm$
Disallow: /store/*ajax.htm$
Disallow: /detail/*Ajax.do$
Disallow: /ajax*.do$
Disallow: /__pc/*
Disallow: /__mobile/*
Disallow: /seo/*
Disallow: /cross-domain/*
Disallow: /*store/group/*
Disallow: /*store/sale-items/*
Disallow: /store/all-wholesale-products/* 
Disallow: /*store/top-rated-products/*
Allow: /i/api/reviews
Disallow: /i/api/*
Disallow: /gcp/300000414/regulatedinformation
Disallow: /img/wsproduct/sku/*

User-agent: YandexBot
Crawl-delay: 0.5
Disallow: /bin/*
Disallow: /search/*
Disallow: /wholesale*
Disallow: /productdetail/*
Disallow: /api/*
Disallow: /api*.do
Disallow: /apps/*
Disallow: /downloads/*
Disallow: /wishlist/*
Disallow: /shopcart/*
Disallow: /brands/*
Disallow: /cp/*
Disallow: /item-img/*
Disallow: /product/*
Disallow: /p4p*list.html
Disallow: /orderList.html
Disallow: /ws*api-server.html$
Disallow: /*mmend.htm$
Disallow: /*Ajax.htm$
Disallow: /*ajax.htm$
Disallow: /store/*ajax.htm$
Disallow: /detail/*Ajax.do$
Disallow: /ajax*.do$
Disallow: /__pc/*
Disallow: /__mobile/*
Disallow: /seo/*
Disallow: /cross-domain/*
Disallow: /*store/group/*
Disallow: /*store/sale-items/*
Disallow: /store/all-wholesale-products/* 
Disallow: /*store/top-rated-products/*
Disallow: /i/api/*
Disallow: /gcp/300000414/regulatedinformation
Disallow: /img/wsproduct/sku/*

Change history

  1. initial snapshot
    • First snapshot of robots.txt archived