Weibo / robots.txt
← back to weibo.com · change detected 15d ago (2026-07-20T19:11:56Z) · modified +15 −35
What changed
- 🤖 AIUnblocked Bytespider (ByteDance) - rules removed
- Removed all rules for user-agent ‘ChatGLM’
- 🤖 AIUnblocked ChatGPT-User (OpenAI) - rules removed
- Removed all rules for user-agent ‘DeepSeek’
- Removed all rules for user-agent ‘DeepSeek Chat’
- Removed all rules for user-agent ‘DeepSeekBot’
- Removed all rules for user-agent ‘Doubao’
- Removed all rules for user-agent ‘Kimi’
- 🤖 AIUnblocked meta-externalagent (Meta) - rules removed
- 🤖 AIUnblocked OAI-SearchBot (OpenAI) - rules removed
- 🤖 AIUnblocked Perplexity-User (Perplexity) - rules removed
- Removed Allow: /ads.txt for *
- Crawl-delay for *: 1 → none
- Crawl-delay for 360Spider: 1 → none
- Crawl-delay for Baiduspider: 1 → none
- Crawl-delay for bingbot: 1 → none
- Crawl-delay for Googlebot: 1 → none
- Crawl-delay for HaosouSpider: 1 → none
- Crawl-delay for smspider: 1 → none
- Crawl-delay for Sogou web spider: 1 → none
- Crawl-delay for YisouSpider: 1 → none
- Removed sitemap https://weibo.com/sitemap.xml
Diff old (2026-07-20T13:08:59Z) → new (2026-07-20T19:11:56Z)
| @@ -1,38 +1,18 @@ | ||
| 1 | - Sitemap: https://weibo.com/sitemap.xml | |
| 2 | - | |
| 3 | 1 | User-Agent: Baiduspider |
| 4 | - User-Agent: 360Spider | |
| 5 | - User-Agent: Googlebot | |
| 6 | - User-Agent: Sogou web spider | |
| 7 | - User-Agent: bingbot | |
| 8 | - User-Agent: smspider | |
| 9 | - User-Agent: HaosouSpider | |
| 10 | - User-Agent: YisouSpider | |
| 11 | - Content-Signal: search=yes, ai-input=yes, ai-train=no | |
| 12 | - Crawl-delay: 1 | |
| 2 | + Disallow: | |
| 3 | + User-agent: 360Spider | |
| 13 | 4 | Disallow: |
| 14 | - | |
| 15 | - User-Agent: Perplexity-User | |
| 16 | - User-Agent: DeepSeek Chat | |
| 17 | - User-Agent: DeepSeekBot | |
| 18 | - User-Agent: Bytespider | |
| 19 | - User-Agent: DeepSeek | |
| 20 | - User-Agent: OAI-SearchBot | |
| 21 | - User-Agent: ChatGPT-User | |
| 22 | - User-Agent: Kimi | |
| 23 | - User-Agent: Doubao | |
| 24 | - User-Agent: meta-externalagent | |
| 25 | - User-Agent: ChatGLM | |
| 26 | - Content-Signal: search=yes, ai-input=yes, ai-train=no | |
| 27 | - Crawl-delay: 1 | |
| 28 | - Allow: /sitemap.xml | |
| 29 | - Allow: /sitemap/ | |
| 30 | - Allow: /2/detail/ | |
| 31 | - Allow: /ttarticle/p/show | |
| 5 | + User-agent: Googlebot | |
| 6 | + Disallow: | |
| 7 | + User-agent: Sogou web spider | |
| 8 | + Disallow: | |
| 9 | + User-agent: bingbot | |
| 10 | + Disallow: | |
| 11 | + User-agent: smspider | |
| 12 | + Disallow: | |
| 13 | + User-agent: HaosouSpider | |
| 14 | + Disallow: | |
| 15 | + User-agent: YisouSpider | |
| 16 | + Disallow: | |
| 17 | + User-agent: * | |
| 32 | 18 | Disallow: / |
| 33 | - | |
| 34 | - User-Agent: * | |
| 35 | - Content-Signal: search=yes, ai-input=yes, ai-train=no | |
| 36 | - Crawl-delay: 1 | |
| 37 | - Disallow: / | |
| 38 | - Allow: /ads.txt |