Corriere della Sera / robots.txt
← back to corriere.it · change detected 1mo ago (2026-06-27T11:10:49Z) · modified +0 −10
What changed
- 🤖 AIUnblocked Bytespider (ByteDance) - rules removed
- Removed all rules for user-agent ‘Diffbot-User’
- Removed all rules for user-agent ‘Scrapy’
- 🤖 AIUnblocked Timpibot (Timpi) - rules removed
- 🤖 AIUnblocked YouBot (You.com) - rules removed
Diff old (2026-06-26T04:48:20Z) → new (2026-06-27T11:10:49Z)
| @@ -25,16 +25,6 @@ | ||
| 25 | 25 | Disallow: / |
| 26 | 26 | User-agent: Yandex |
| 27 | 27 | Disallow: / # prohibits crawling for the entire site |
| 28 | - User-agent: Bytespider | |
| 29 | - Disallow: / | |
| 30 | - User-agent: Scrapy | |
| 31 | - Disallow: / | |
| 32 | - User-agent: Diffbot-User | |
| 33 | - Disallow: / | |
| 34 | - User-agent: YouBot | |
| 35 | - Disallow: / | |
| 36 | - User-agent: Timpibot | |
| 37 | - Disallow: / | |
| 38 | 28 | # Fine Bot Exclusion Protocol |
| 39 | 29 | |
| 40 | 30 | User-agent: * |