Weibo / robots.txt
← back to weibo.com · change detected 15d ago (2026-07-20T13:08:59Z) · modified +35 −15
What changed
- 🤖 AIBlocked Bytespider (ByteDance) entirely
- Blocked user-agent ‘ChatGLM’ entirely
- 🤖 AIBlocked ChatGPT-User (OpenAI) entirely
- Blocked user-agent ‘DeepSeek’ entirely
- Blocked user-agent ‘DeepSeek Chat’ entirely
- Blocked user-agent ‘DeepSeekBot’ entirely
- Blocked user-agent ‘Doubao’ entirely
- Blocked user-agent ‘Kimi’ entirely
- 🤖 AIBlocked meta-externalagent (Meta) entirely
- 🤖 AIBlocked OAI-SearchBot (OpenAI) entirely
- 🤖 AIBlocked Perplexity-User (Perplexity) entirely
- Added Allow: /ads.txt for *
- Crawl-delay for *: none → 1
- Crawl-delay for 360Spider: none → 1
- Crawl-delay for Baiduspider: none → 1
- Crawl-delay for bingbot: none → 1
- Crawl-delay for Googlebot: none → 1
- Crawl-delay for HaosouSpider: none → 1
- Crawl-delay for smspider: none → 1
- Crawl-delay for Sogou web spider: none → 1
- Crawl-delay for YisouSpider: none → 1
- Added sitemap https://weibo.com/sitemap.xml
Diff old (2026-06-20T01:10:30Z) → new (2026-07-20T13:08:59Z)
| @@ -1,18 +1,38 @@ | ||
| 1 | + Sitemap: https://weibo.com/sitemap.xml | |
| 2 | + | |
| 1 | 3 | User-Agent: Baiduspider |
| 2 | - Disallow: | |
| 3 | - User-agent: 360Spider | |
| 4 | + User-Agent: 360Spider | |
| 5 | + User-Agent: Googlebot | |
| 6 | + User-Agent: Sogou web spider | |
| 7 | + User-Agent: bingbot | |
| 8 | + User-Agent: smspider | |
| 9 | + User-Agent: HaosouSpider | |
| 10 | + User-Agent: YisouSpider | |
| 11 | + Content-Signal: search=yes, ai-input=yes, ai-train=no | |
| 12 | + Crawl-delay: 1 | |
| 4 | 13 | Disallow: |
| 5 | - User-agent: Googlebot | |
| 6 | - Disallow: | |
| 7 | - User-agent: Sogou web spider | |
| 8 | - Disallow: | |
| 9 | - User-agent: bingbot | |
| 10 | - Disallow: | |
| 11 | - User-agent: smspider | |
| 12 | - Disallow: | |
| 13 | - User-agent: HaosouSpider | |
| 14 | - Disallow: | |
| 15 | - User-agent: YisouSpider | |
| 16 | - Disallow: | |
| 17 | - User-agent: * | |
| 14 | + | |
| 15 | + User-Agent: Perplexity-User | |
| 16 | + User-Agent: DeepSeek Chat | |
| 17 | + User-Agent: DeepSeekBot | |
| 18 | + User-Agent: Bytespider | |
| 19 | + User-Agent: DeepSeek | |
| 20 | + User-Agent: OAI-SearchBot | |
| 21 | + User-Agent: ChatGPT-User | |
| 22 | + User-Agent: Kimi | |
| 23 | + User-Agent: Doubao | |
| 24 | + User-Agent: meta-externalagent | |
| 25 | + User-Agent: ChatGLM | |
| 26 | + Content-Signal: search=yes, ai-input=yes, ai-train=no | |
| 27 | + Crawl-delay: 1 | |
| 28 | + Allow: /sitemap.xml | |
| 29 | + Allow: /sitemap/ | |
| 30 | + Allow: /2/detail/ | |
| 31 | + Allow: /ttarticle/p/show | |
| 18 | 32 | Disallow: / |
| 33 | + | |
| 34 | + User-Agent: * | |
| 35 | + Content-Signal: search=yes, ai-input=yes, ai-train=no | |
| 36 | + Crawl-delay: 1 | |
| 37 | + Disallow: / | |
| 38 | + Allow: /ads.txt |