The New Yorker / robots.txt
← back to newyorker.com · change detected 11d ago (2026-09-08T12:01:45Z) · modified +35 −26
What changed
- Blocked user-agent ‘Bravebot’ entirely
- Blocked user-agent ‘Brightbot 1.0’ entirely
- Blocked user-agent ‘factset_spyderbot’ entirely
- Blocked user-agent ‘FirecrawlAgent’ entirely
- Blocked user-agent ‘Scrapy’ entirely
- Blocked user-agent ‘Shap-User’ entirely
- Blocked user-agent ‘ShapBot’ entirely
- Blocked user-agent ‘Yisouspider’ entirely
- 🤖 AIBlocked YouBot (You.com) entirely
Diff old (2026-07-29T16:38:54Z) → new (2026-09-08T12:01:45Z)
| @@ -39,43 +39,52 @@ | ||
| 39 | 39 | Disallow: /newsletters/ |
| 40 | 40 | ### |
| 41 | 41 | |
| 42 | + User-agent: AmazonAdBot | |
| 42 | 43 | User-agent: FacebookExternalHit |
| 44 | + User-agent: Google-InspectionTool | |
| 45 | + User-agent: LinkedInBot | |
| 46 | + User-agent: Pinterestbot | |
| 47 | + User-agent: Storebot-Google | |
| 48 | + User-agent: TikTokSpider | |
| 43 | 49 | User-agent: Twitterbot |
| 44 | - User-agent: Pinterestbot | |
| 45 | - User-agent: LinkedInBot | |
| 46 | - User-agent: TikTokSpider | |
| 47 | - User-agent: Storebot-Google | |
| 48 | - User-agent: Google-InspectionTool | |
| 49 | - User-agent: AmazonAdBot | |
| 50 | 50 | Allow: / |
| 51 | 51 | |
| 52 | + User-agent: Amazonbot | |
| 53 | + User-agent: Applebot-Extended | |
| 54 | + User-agent: archive.org_bot | |
| 55 | + User-agent: Bravebot | |
| 56 | + User-agent: Brightbot 1.0 | |
| 57 | + User-agent: Bytespider | |
| 58 | + User-agent: CCBot | |
| 59 | + User-agent: Claude-SearchBot | |
| 60 | + User-agent: Claude-User | |
| 61 | + User-agent: ClaudeBot | |
| 62 | + User-agent: cohere-ai | |
| 63 | + User-agent: cohere-training-data-crawler | |
| 64 | + User-agent: Diffbot | |
| 65 | + User-agent: DuckAssistBot | |
| 66 | + User-agent: factset_spyderbot | |
| 67 | + User-agent: FirecrawlAgent | |
| 68 | + User-agent: Google-CloudVertexBot | |
| 52 | 69 | User-agent: Google-Extended |
| 53 | - User-agent: Google-CloudVertexBot | |
| 54 | 70 | User-agent: GoogleOther |
| 55 | - User-agent: Applebot-Extended | |
| 56 | - User-agent: Amazonbot | |
| 57 | - User-agent: meta-webindexer | |
| 71 | + User-agent: heritrix | |
| 72 | + User-agent: ia_archiver | |
| 73 | + User-agent: ia_archiver-web.archive.org | |
| 58 | 74 | User-agent: meta-externalagent |
| 59 | 75 | User-agent: meta-externalfetcher |
| 60 | - User-agent: ClaudeBot | |
| 61 | - User-agent: Claude-SearchBot | |
| 62 | - User-agent: Claude-User | |
| 76 | + User-agent: meta-webindexer | |
| 77 | + User-agent: MistralAI-User | |
| 78 | + User-agent: PanguBot | |
| 79 | + User-agent: Perplexity-User | |
| 63 | 80 | User-agent: PerplexityBot |
| 64 | - User-agent: Perplexity-User | |
| 65 | - User-agent: cohere-training-data-crawler | |
| 66 | - User-agent: cohere-ai | |
| 67 | - User-agent: CCBot | |
| 68 | - User-agent: PanguBot | |
| 69 | 81 | User-agent: PetalBot |
| 70 | - User-agent: Bytespider | |
| 71 | - User-agent: Diffbot | |
| 72 | - User-agent: DuckAssistBot | |
| 73 | - User-agent: MistralAI-User | |
| 82 | + User-agent: Scrapy | |
| 83 | + User-agent: Shap-User | |
| 84 | + User-agent: ShapBot | |
| 74 | 85 | User-agent: Timpibot |
| 86 | + User-agent: Yisouspider | |
| 87 | + User-agent: YouBot | |
| 75 | 88 | User-agent: Webzio |
| 76 | 89 | User-agent: Webzio-Extended |
| 77 | - User-agent: archive.org_bot | |
| 78 | - User-agent: ia_archiver | |
| 79 | - User-agent: ia_archiver-web.archive.org | |
| 80 | - User-agent: heritrix | |
| 81 | 90 | Disallow: / |