Der Spiegel / robots.txt
← back to spiegel.de · change detected 2mo ago (2026-09-08T12:01:45Z) · modified +18 −0
What changed
- Blocked user-agent ‘archive.org_bot’ entirely
- Blocked user-agent ‘heritrix’ entirely
- Blocked user-agent ‘ia_archiver’ entirely
- Blocked user-agent ‘ia_archiver-web.archive.org’ entirely
- Blocked user-agent ‘ia_archiver2’ entirely
- Blocked user-agent ‘Special_Archiver’ entirely
- Blocked user-agent ‘WaybackSavePage’ entirely
Diff old (2026-08-11T15:08:50Z) → new (2026-09-08T12:01:45Z)
| @@ -223,6 +223,24 @@ | ||
| 223 | 223 | |
| 224 | 224 | |
| 225 | 225 | # ========================================== |
| 226 | + # ARCHIV-CRAWLER (SPERRE) | |
| 227 | + # ========================================== | |
| 228 | + User-agent: archive.org_bot | |
| 229 | + Disallow: / | |
| 230 | + User-agent: WaybackSavePage | |
| 231 | + Disallow: / | |
| 232 | + User-agent: heritrix | |
| 233 | + Disallow: / | |
| 234 | + User-agent: Special_Archiver | |
| 235 | + Disallow: / | |
| 236 | + User-agent: ia_archiver-web.archive.org | |
| 237 | + Disallow: / | |
| 238 | + User-agent: ia_archiver2 | |
| 239 | + Disallow: / | |
| 240 | + User-agent: ia_archiver | |
| 241 | + Disallow: / | |
| 242 | + | |
| 243 | + # ========================================== | |
| 226 | 244 | # HISTORISCHE BOTS & MASSEN-DOWNLOADER |
| 227 | 245 | # ========================================== |
| 228 | 246 | User-agent: Bloodhound |