Corriere della Sera / robots.txt
← back to corriere.it · change detected 1mo ago (2026-06-27T17:18:59Z) · modified +10 −0
What changed
- 🤖 AIBlocked Bytespider (ByteDance) entirely
- Blocked user-agent ‘Diffbot-User’ entirely
- Blocked user-agent ‘Scrapy’ entirely
- 🤖 AIBlocked Timpibot (Timpi) entirely
- 🤖 AIBlocked YouBot (You.com) entirely
Diff old (2026-06-27T11:10:49Z) → new (2026-06-27T17:18:59Z)
| @@ -25,6 +25,16 @@ | ||
| 25 | 25 | Disallow: / |
| 26 | 26 | User-agent: Yandex |
| 27 | 27 | Disallow: / # prohibits crawling for the entire site |
| 28 | + User-agent: Bytespider | |
| 29 | + Disallow: / | |
| 30 | + User-agent: Scrapy | |
| 31 | + Disallow: / | |
| 32 | + User-agent: Diffbot-User | |
| 33 | + Disallow: / | |
| 34 | + User-agent: YouBot | |
| 35 | + Disallow: / | |
| 36 | + User-agent: Timpibot | |
| 37 | + Disallow: / | |
| 28 | 38 | # Fine Bot Exclusion Protocol |
| 29 | 39 | |
| 30 | 40 | User-agent: * |