Wired / robots.txt
← back to wired.com · change detected 11d ago (2026-09-08T12:01:45Z) · modified +35 −26
What changed
- Blocked user-agent ‘Bravebot’ entirely
- Blocked user-agent ‘Brightbot 1.0’ entirely
- Blocked user-agent ‘factset_spyderbot’ entirely
- Blocked user-agent ‘FirecrawlAgent’ entirely
- Blocked user-agent ‘Scrapy’ entirely
- Blocked user-agent ‘Shap-User’ entirely
- Blocked user-agent ‘ShapBot’ entirely
- Blocked user-agent ‘Yisouspider’ entirely
- 🤖 AIBlocked YouBot (You.com) entirely
Diff old (2026-07-29T16:38:54Z) → new (2026-09-08T12:01:45Z)
| @@ -40,43 +40,52 @@ | ||
| 40 | 40 | Disallow: */testing-products- |
| 41 | 41 | ### |
| 42 | 42 | |
| 43 | + User-agent: AmazonAdBot | |
| 43 | 44 | User-agent: FacebookExternalHit |
| 45 | + User-agent: Google-InspectionTool | |
| 46 | + User-agent: LinkedInBot | |
| 47 | + User-agent: Pinterestbot | |
| 48 | + User-agent: Storebot-Google | |
| 49 | + User-agent: TikTokSpider | |
| 44 | 50 | User-agent: Twitterbot |
| 45 | - User-agent: Pinterestbot | |
| 46 | - User-agent: LinkedInBot | |
| 47 | - User-agent: TikTokSpider | |
| 48 | - User-agent: Storebot-Google | |
| 49 | - User-agent: Google-InspectionTool | |
| 50 | - User-agent: AmazonAdBot | |
| 51 | 51 | Allow: / |
| 52 | 52 | |
| 53 | + User-agent: Amazonbot | |
| 54 | + User-agent: Applebot-Extended | |
| 55 | + User-agent: archive.org_bot | |
| 56 | + User-agent: Bravebot | |
| 57 | + User-agent: Brightbot 1.0 | |
| 58 | + User-agent: Bytespider | |
| 59 | + User-agent: CCBot | |
| 60 | + User-agent: Claude-SearchBot | |
| 61 | + User-agent: Claude-User | |
| 62 | + User-agent: ClaudeBot | |
| 63 | + User-agent: cohere-ai | |
| 64 | + User-agent: cohere-training-data-crawler | |
| 65 | + User-agent: Diffbot | |
| 66 | + User-agent: DuckAssistBot | |
| 67 | + User-agent: factset_spyderbot | |
| 68 | + User-agent: FirecrawlAgent | |
| 69 | + User-agent: Google-CloudVertexBot | |
| 53 | 70 | User-agent: Google-Extended |
| 54 | - User-agent: Google-CloudVertexBot | |
| 55 | 71 | User-agent: GoogleOther |
| 56 | - User-agent: Applebot-Extended | |
| 57 | - User-agent: Amazonbot | |
| 58 | - User-agent: meta-webindexer | |
| 72 | + User-agent: heritrix | |
| 73 | + User-agent: ia_archiver | |
| 74 | + User-agent: ia_archiver-web.archive.org | |
| 59 | 75 | User-agent: meta-externalagent |
| 60 | 76 | User-agent: meta-externalfetcher |
| 61 | - User-agent: ClaudeBot | |
| 62 | - User-agent: Claude-SearchBot | |
| 63 | - User-agent: Claude-User | |
| 77 | + User-agent: meta-webindexer | |
| 78 | + User-agent: MistralAI-User | |
| 79 | + User-agent: PanguBot | |
| 80 | + User-agent: Perplexity-User | |
| 64 | 81 | User-agent: PerplexityBot |
| 65 | - User-agent: Perplexity-User | |
| 66 | - User-agent: cohere-training-data-crawler | |
| 67 | - User-agent: cohere-ai | |
| 68 | - User-agent: CCBot | |
| 69 | - User-agent: PanguBot | |
| 70 | 82 | User-agent: PetalBot |
| 71 | - User-agent: Bytespider | |
| 72 | - User-agent: Diffbot | |
| 73 | - User-agent: DuckAssistBot | |
| 74 | - User-agent: MistralAI-User | |
| 83 | + User-agent: Scrapy | |
| 84 | + User-agent: Shap-User | |
| 85 | + User-agent: ShapBot | |
| 75 | 86 | User-agent: Timpibot |
| 87 | + User-agent: Yisouspider | |
| 88 | + User-agent: YouBot | |
| 76 | 89 | User-agent: Webzio |
| 77 | 90 | User-agent: Webzio-Extended |
| 78 | - User-agent: archive.org_bot | |
| 79 | - User-agent: ia_archiver | |
| 80 | - User-agent: ia_archiver-web.archive.org | |
| 81 | - User-agent: heritrix | |
| 82 | 91 | Disallow: / |