Newsweek / robots.txt
← back to newsweek.com · change detected 8d ago (2026-10-29T23:30:26Z) · modified +6 −2
What changed
- 🤖 AIBlocked Perplexity-User (Perplexity) entirely
- Blocked user-agent ‘Shap-User’ entirely
- Blocked user-agent ‘ShapBot’ entirely
- 🤖 AIUnblocked Amazonbot (Amazon) - rules removed
Diff old (2026-10-23T03:13:22Z) → new (2026-10-29T23:30:26Z)
| @@ -8,8 +8,6 @@ | ||
| 8 | 8 | Disallow: /_fs-ch-1T1wmsGaOgGaSxcX/ |
| 9 | 9 | |
| 10 | 10 | # Disallow Bots |
| 11 | - User-agent: Amazonbot | |
| 12 | - Disallow: / | |
| 13 | 11 | User-agent: anthropic-ai |
| 14 | 12 | Disallow: / |
| 15 | 13 | User-agent: Applebot-Extended |
| @@ -54,7 +52,13 @@ | ||
| 54 | 52 | Disallow: / |
| 55 | 53 | User-agent: PerplexityBot |
| 56 | 54 | Disallow: / |
| 55 | + User-agent: Perplexity-User | |
| 56 | + Disallow: / | |
| 57 | 57 | User-agent: Scrapy |
| 58 | + Disallow: / | |
| 59 | + User-agent: ShapBot | |
| 60 | + Disallow: / | |
| 61 | + User-agent: Shap-User | |
| 58 | 62 | Disallow: / |
| 59 | 63 | User-agent: TurnitinBot |
| 60 | 64 | Disallow: / |