The Atlantic / robots.txt
← back to theatlantic.com · change detected 1d ago (2026-09-17T21:12:52Z) · modified +5 −1
What changed
- Unblocked user-agent ‘Applebot’
- Added Allow: / for Applebot
- 🤖 AIAdded Allow: /sponsored/ for Claude-SearchBot
- 🤖 AIAdded Allow: /sponsored/ for Claude-User
- 🤖 AIAdded Allow: /sponsored/ for Perplexity-User
- 🤖 AIAdded Allow: /sponsored/ for PerplexityBot
Diff old (2026-06-20T01:10:30Z) → new (2026-09-17T21:12:52Z)
| @@ -48,7 +48,7 @@ | ||
| 48 | 48 | Disallow: / |
| 49 | 49 | |
| 50 | 50 | User-agent: Applebot |
| 51 | - Disallow: / | |
| 51 | + Allow: / | |
| 52 | 52 | |
| 53 | 53 | User-agent: Applebot-Extended |
| 54 | 54 | Disallow: / |
| @@ -77,9 +77,11 @@ | ||
| 77 | 77 | Disallow: / |
| 78 | 78 | |
| 79 | 79 | User-agent: Claude-SearchBot |
| 80 | + Allow: /sponsored/ | |
| 80 | 81 | Disallow: / |
| 81 | 82 | |
| 82 | 83 | User-agent: Claude-User |
| 84 | + Allow: /sponsored/ | |
| 83 | 85 | Disallow: / |
| 84 | 86 | |
| 85 | 87 | User-agent: Claude-Web |
| @@ -147,9 +149,11 @@ | ||
| 147 | 149 | Disallow: / |
| 148 | 150 | |
| 149 | 151 | User-agent: PerplexityBot |
| 152 | + Allow: /sponsored/ | |
| 150 | 153 | Disallow: / |
| 151 | 154 | |
| 152 | 155 | User-agent: Perplexity-User |
| 156 | + Allow: /sponsored/ | |
| 153 | 157 | Disallow: / |
| 154 | 158 | |
| 155 | 159 | User-agent: ProRataInc |