Corriere della Sera / robots.txt
← back to corriere.it · change detected 8d ago (2026-09-10T18:52:44Z) · modified +15 −0
What changed
- Added rules for user-agent ‘Amzn-SearchBot’
- 🤖 AIBlocked ChatGPT-User (OpenAI) entirely
- 🤖 AIBlocked Claude-Web (Anthropic) entirely
- 🤖 AIAdded rules for AI crawler OAI-SearchBot (OpenAI)
- 🤖 AIAdded rules for AI crawler PerplexityBot (Perplexity)
- Added Allow: / for *
- Added Allow: /nativeadv/ for AppleBot
Diff old (2026-08-07T13:42:41Z) → new (2026-09-10T18:52:44Z)
| @@ -11,6 +11,7 @@ | ||
| 11 | 11 | Disallow: / |
| 12 | 12 | User-agent: AppleBot |
| 13 | 13 | Disallow: / |
| 14 | + Allow: /nativeadv/ | |
| 14 | 15 | User-agent: Google-Extended |
| 15 | 16 | Disallow: / |
| 16 | 17 | User-agent: Applebot-Extended |
| @@ -19,8 +20,20 @@ | ||
| 19 | 20 | Disallow: / |
| 20 | 21 | User-agent: Amazonbot |
| 21 | 22 | Disallow: / |
| 23 | + User-agent: Amzn-SearchBot | |
| 24 | + Allow: / | |
| 22 | 25 | User-agent: Meta-ExternalAgent |
| 23 | 26 | Disallow: / |
| 27 | + User-agent: OAI-SearchBot | |
| 28 | + Allow: / | |
| 29 | + User-agent: ChatGPT-User | |
| 30 | + Disallow: / | |
| 31 | + Allow: /nativeadv/ | |
| 32 | + User-agent: Claude-Web | |
| 33 | + Disallow: / | |
| 34 | + Allow: /nativeadv/ | |
| 35 | + User-agent: PerplexityBot | |
| 36 | + Allow: / | |
| 24 | 37 | User-agent: PetalBot |
| 25 | 38 | Disallow: / |
| 26 | 39 | User-agent: Yandex |
| @@ -44,6 +57,8 @@ | ||
| 44 | 57 | # Fine Bot Exclusion Protocol |
| 45 | 58 | |
| 46 | 59 | User-agent: * |
| 60 | + Allow: / | |
| 61 | + Content-Signal: search=yes, ai-train=yes, ai-input=no | |
| 47 | 62 | Disallow: *{{data.c.link_offerId}} |
| 48 | 63 | |
| 49 | 64 |