China Mobile / robots.txt snapshot
← back to chinamobileltd.com · fetched 2026-08-31T03:28:12Z (2mo ago) · HTTP 200 · 1658 bytes · sha256 ce64db026ed7fd15 · raw
final URL: https://www.chinamobileltd.com//robots.txt
| 1 | # ========================================== |
| 2 | # 1. 允許 SEO & AEO (主要 AI 搜尋引擎) 爬蟲 |
| 3 | # ========================================== |
| 4 | User-agent: Googlebot |
| 5 | User-agent: Bingbot |
| 6 | User-agent: Baiduspider |
| 7 | |
| 8 | # AEO / 主流 AI Search Engine & DeepSeek Crawlers |
| 9 | User-agent: DeepSeekBot # DeepSeek AI 爬蟲 |
| 10 | User-agent: DeepSeek # DeepSeek 備用識別標籤 |
| 11 | User-agent: OAI-SearchBot # OpenAI SearchBot (ChatGPT 搜尋功能) |
| 12 | User-agent: ChatGPT-User # ChatGPT 瀏覽網頁功能 |
| 13 | User-agent: PerplexityBot # Perplexity AI 搜尋 |
| 14 | User-agent: Claude-Web # Claude 網頁即時搜尋 |
| 15 | User-agent: Amazonbot # Amazon / Alexa 搜尋與回答 |
| 16 | |
| 17 | Allow: / |
| 18 | |
| 19 | # ========================================== |
| 20 | # 2. 封鎖 AI 模型訓練/數據採集 Bot |
| 21 | # ========================================== |
| 22 | User-agent: GPTBot # OpenAI 數據訓練 |
| 23 | User-agent: Google-Extended # Google Gemini 模型訓練 |
| 24 | User-agent: CCBot # Common Crawl (大量 AI 訓練數據源) |
| 25 | User-agent: Anthropic-AI # Anthropic 模型訓練 |
| 26 | User-agent: ClaudeBot # Claude 模型訓練 |
| 27 | User-agent: Bytespider # 字節跳動 (TikTok) 爬蟲 |
| 28 | User-agent: Diffbot # 數據抽取與 AI 訓練 |
| 29 | User-agent: Applebot-Extended # Apple AI 模型訓練 |
| 30 | User-agent: FacebookBot # Meta 數據採集 |
| 31 | User-agent: cohere-ai # Cohere AI 訓練 |
| 32 | |
| 33 | Disallow: / |
| 34 | |
| 35 | # ========================================== |
| 36 | # 3. 封鎖特定地區搜尋引擎、小眾 Bot & 競品分析爬蟲 |
| 37 | # ========================================== |
| 38 | # 封鎖 You.com (小眾 AI 搜尋) |
| 39 | User-agent: YouBot |
| 40 | |
| 41 | # 封鎖 Yandex (俄羅斯搜尋引擎) |
| 42 | User-agent: Yandex |
| 43 | User-agent: YandexBot |
| 44 | |
| 45 | # 封鎖 DuckDuckGo (隱私搜尋引擎) |
| 46 | User-agent: DuckDuckBot |
| 47 | |
| 48 | # 封鎖 SEO 工具爬蟲 |
| 49 | User-agent: AhrefsBot |
| 50 | User-agent: MJ12bot |
| 51 | User-agent: SemrushBot |
| 52 | User-agent: DotBot |
| 53 | User-agent: Rogerbot |
| 54 | |
| 55 | Disallow: / |