Der Spiegel / robots.txt
← back to spiegel.de · change detected 1mo ago (2026-08-11T15:08:50Z) · modified +209 −86
What changed
- Blocked user-agent ‘AhrefsSiteAudit’ entirely
- 🤖 AIBlocked AI2Bot (Allen Institute for AI) entirely
- 🤖 AIBlocked Amazonbot (Amazon) entirely
- Blocked user-agent ‘audisto’ entirely
- Blocked user-agent ‘audisto-essential’ entirely
- Blocked user-agent ‘barkrowler’ entirely
- Blocked user-agent ‘bigsur.ai’ entirely
- Blocked user-agent ‘Bloodhound’ entirely
- Blocked user-agent ‘botify’ entirely
- 🤖 AIBlocked Claude-SearchBot (Anthropic) entirely
- 🤖 AIBlocked Claude-User (Anthropic) entirely
- Blocked user-agent ‘CloudVertexBot’ entirely
- Blocked user-agent ‘Cocolyzebot’ entirely
- Blocked user-agent ‘cohere-training-data-collector’ entirely
- Blocked user-agent ‘cydralspider’ entirely
- Blocked user-agent ‘deepcrawl’ entirely
- Blocked user-agent ‘DeepL-Bot’ entirely
- Blocked user-agent ‘DeepL-Translate’ entirely
- Blocked user-agent ‘DeepSeek’ entirely
- Blocked user-agent ‘DeepSeekBot’ entirely
- Blocked user-agent ‘Devin’ entirely
- Blocked user-agent ‘downloadexpress’ entirely
- Blocked user-agent ‘Ezooms’ entirely
- Blocked user-agent ‘Fasterfox’ entirely
- Blocked user-agent ‘gammaSpider’ entirely
- Blocked user-agent ‘HTTPBannerDetection’ entirely
- Blocked user-agent ‘iaskspider’ entirely
- Blocked user-agent ‘Kangaroo Bot’ entirely
- Blocked user-agent ‘Landau-Media-Spider’ entirely
- Blocked user-agent ‘LinerBot’ entirely
- Blocked user-agent ‘lumar’ entirely
- Blocked user-agent ‘MegaIndex’ entirely
- Blocked user-agent ‘Meltwater’ entirely
- 🤖 AIBlocked MistralAI-User (Mistral AI) entirely
- Blocked user-agent ‘MozDotNet’ entirely
- Blocked user-agent ‘NewsNow’ entirely
- Blocked user-agent ‘ObjectsSearch’ entirely
- Blocked user-agent ‘oncrawl’ entirely
- 🤖 AIBlocked PanguBot (Huawei) entirely
- Blocked user-agent ‘PetalBot’ entirely
- Blocked user-agent ‘Pimptrain’ entirely
- Blocked user-agent ‘Raven’ entirely
- Blocked user-agent ‘rogerbot’ entirely
- Blocked user-agent ‘RyteBot’ entirely
- Blocked user-agent ‘Scrapy’ entirely
- Blocked user-agent ‘Searchmetrics’ entirely
- Blocked user-agent ‘SearchmetricsBot’ entirely
- Blocked user-agent ‘SemrushBot-BA’ entirely
- Blocked user-agent ‘SemrushBot-SA’ entirely
- Blocked user-agent ‘SemrushBot-SI’ entirely
- ... and 21 more change(s)
Diff old (2026-06-20T01:10:30Z) → new (2026-08-11T15:08:50Z)
| @@ -20,6 +20,7 @@ | ||
| 20 | 20 | Allow: /reise/ |
| 21 | 21 | Allow: /psychologie/ |
| 22 | 22 | Allow: /stil/ |
| 23 | + Allow: /tests/ | |
| 23 | 24 | Disallow: / |
| 24 | 25 | |
| 25 | 26 | # TLP-6507: Testweise Freischaltung der OpenAI-Suchcrawler fuer ausgewaehlte Bereiche |
| @@ -31,107 +32,229 @@ | ||
| 31 | 32 | Allow: /reise/ |
| 32 | 33 | Allow: /psychologie/ |
| 33 | 34 | Allow: /stil/ |
| 35 | + Allow: /tests/ | |
| 34 | 36 | Disallow: / |
| 35 | 37 | |
| 38 | + # ========================================== | |
| 39 | + # KI-CRAWLER & KI-TRAINING (SPERRE) | |
| 40 | + # ========================================== | |
| 41 | + User-agent: GPTBot | |
| 42 | + Disallow: / | |
| 43 | + User-agent: anthropic-ai | |
| 44 | + Disallow: / | |
| 45 | + User-agent: Claude-Web | |
| 46 | + Disallow: / | |
| 47 | + User-agent: ClaudeBot | |
| 48 | + Disallow: / | |
| 49 | + User-agent: Claude-SearchBot | |
| 50 | + Disallow: / | |
| 51 | + User-agent: Claude-User | |
| 52 | + Disallow: / | |
| 53 | + User-agent: CloudVertexBot | |
| 54 | + Disallow: / | |
| 36 | 55 | User-agent: cohere-ai |
| 37 | 56 | Disallow: / |
| 38 | - | |
| 39 | 57 | User-agent: cohere-training-data-crawler |
| 40 | 58 | Disallow: / |
| 59 | + User-agent: cohere-training-data-collector | |
| 60 | + Disallow: / | |
| 61 | + User-agent: DeepSeekBot | |
| 62 | + Disallow: / | |
| 63 | + User-agent: DeepSeek | |
| 64 | + Disallow: / | |
| 65 | + User-agent: Meta-ExternalAgent | |
| 66 | + Disallow: / | |
| 67 | + User-agent: FacebookBot | |
| 68 | + Disallow: / | |
| 69 | + User-agent: Applebot-Extended | |
| 70 | + Disallow: / | |
| 71 | + User-agent: MistralAI-User | |
| 72 | + Disallow: / | |
| 73 | + User-agent: Amazonbot | |
| 74 | + Disallow: / | |
| 75 | + User-agent: AI2Bot | |
| 76 | + Disallow: / | |
| 77 | + User-agent: Kangaroo Bot | |
| 78 | + Disallow: / | |
| 79 | + User-agent: PanguBot | |
| 80 | + Disallow: / | |
| 81 | + User-agent: PetalBot | |
| 82 | + Disallow: / | |
| 83 | + User-agent: Devin | |
| 84 | + Disallow: / | |
| 85 | + User-agent: bigsur.ai | |
| 86 | + Disallow: / | |
| 87 | + User-agent: LinerBot | |
| 88 | + Disallow: / | |
| 89 | + User-agent: CCBot | |
| 90 | + Disallow: / | |
| 91 | + User-agent: YouBot | |
| 92 | + Disallow: / | |
| 93 | + User-agent: YouBot-Search | |
| 94 | + Disallow: / | |
| 95 | + User-agent: iaskspider | |
| 96 | + Disallow: / | |
| 97 | + User-agent: YoudaoBot | |
| 98 | + Disallow: / | |
| 99 | + User-agent: DeepL-Translate | |
| 100 | + Disallow: / | |
| 101 | + User-agent: DeepL-Bot | |
| 102 | + Disallow: / | |
| 41 | 103 | |
| 104 | + # ========================================== | |
| 105 | + # SEO-CRAWLER (SPERRE) | |
| 106 | + # ========================================== | |
| 107 | + User-agent: AhrefsBot | |
| 108 | + Disallow: / | |
| 109 | + User-agent: AhrefsSiteAudit | |
| 110 | + Disallow: / | |
| 111 | + User-agent: SemrushBot | |
| 112 | + Disallow: / | |
| 113 | + User-agent: SemrushBot-SA | |
| 114 | + Disallow: / | |
| 115 | + User-agent: SemrushBot-BA | |
| 116 | + Disallow: / | |
| 117 | + User-agent: SemrushBot-SI | |
| 118 | + Disallow: / | |
| 119 | + User-agent: SemrushBot-SWA | |
| 120 | + Disallow: / | |
| 121 | + User-agent: SiteAuditBot | |
| 122 | + Disallow: / | |
| 123 | + User-agent: SplitSignalBot | |
| 124 | + Disallow: / | |
| 125 | + User-agent: xovi | |
| 126 | + Disallow: / | |
| 127 | + User-agent: XoviBot | |
| 128 | + Disallow: / | |
| 129 | + User-agent: Seobility | |
| 130 | + Disallow: / | |
| 131 | + User-agent: SeobilityBot | |
| 132 | + Disallow: / | |
| 133 | + User-agent: RyteBot | |
| 134 | + Disallow: / | |
| 135 | + User-agent: SEOkicks | |
| 136 | + Disallow: / | |
| 137 | + User-agent: SEOkicks-Robot | |
| 138 | + Disallow: / | |
| 139 | + User-agent: Searchmetrics | |
| 140 | + Disallow: / | |
| 141 | + User-agent: SearchmetricsBot | |
| 142 | + Disallow: / | |
| 143 | + User-agent: audisto | |
| 144 | + Disallow: / | |
| 145 | + User-agent: audisto-essential | |
| 146 | + Disallow: / | |
| 147 | + User-agent: MJ12bot | |
| 148 | + Disallow: / | |
| 149 | + User-agent: dotbot | |
| 150 | + Disallow: / | |
| 151 | + User-agent: rogerbot | |
| 152 | + Disallow: / | |
| 153 | + User-agent: Ezooms | |
| 154 | + Disallow: / | |
| 155 | + User-agent: barkrowler | |
| 156 | + Disallow: / | |
| 157 | + User-agent: serpstatbot | |
| 158 | + Disallow: / | |
| 159 | + User-agent: BLEXBot | |
| 160 | + Disallow: / | |
| 161 | + User-agent: DataForSeoBot | |
| 162 | + Disallow: / | |
| 163 | + User-agent: MegaIndex | |
| 164 | + Disallow: / | |
| 165 | + User-agent: lumar | |
| 166 | + Disallow: / | |
| 167 | + User-agent: deepcrawl | |
| 168 | + Disallow: / | |
| 169 | + User-agent: oncrawl | |
| 170 | + Disallow: / | |
| 171 | + User-agent: sitebulb | |
| 172 | + Disallow: / | |
| 173 | + User-agent: botify | |
| 174 | + Disallow: / | |
| 175 | + User-agent: Siteimprove | |
| 176 | + Disallow: / | |
| 177 | + User-agent: Siteimprove Crawl | |
| 178 | + Disallow: / | |
| 179 | + User-agent: MozDotNet | |
| 180 | + Disallow: / | |
| 181 | + User-agent: Cocolyzebot | |
| 182 | + Disallow: / | |
| 183 | + User-agent: Raven | |
| 184 | + Disallow: / | |
| 185 | + | |
| 186 | + # ========================================== | |
| 187 | + # MEDIENBEOBACHTUNG, AGGREGATOREN & SCRAPER | |
| 188 | + # ========================================== | |
| 189 | + User-agent: Meltwater | |
| 190 | + Disallow: / | |
| 191 | + User-agent: NewsNow | |
| 192 | + Disallow: / | |
| 42 | 193 | User-agent: Webzio-Extended |
| 43 | 194 | Disallow: / |
| 44 | - | |
| 45 | - User-agent: YouBot | |
| 195 | + User-agent: magpie-crawler | |
| 196 | + Disallow: / | |
| 197 | + User-agent: omgili | |
| 198 | + Disallow: / | |
| 199 | + User-agent: omgilibot | |
| 200 | + Disallow: / | |
| 201 | + User-agent: Baiduspider | |
| 202 | + Disallow: / | |
| 203 | + User-agent: Yeti | |
| 204 | + Disallow: / | |
| 205 | + User-agent: sentibot | |
| 206 | + Disallow: / | |
| 207 | + User-agent: Bytespider | |
| 208 | + Disallow: / | |
| 209 | + User-agent: SirdataBot | |
| 210 | + Disallow: / | |
| 211 | + User-agent: LCC | |
| 212 | + Disallow: / | |
| 213 | + User-agent: TurnitinBot | |
| 214 | + Disallow: / | |
| 215 | + User-agent: ImagesiftBot | |
| 216 | + Disallow: / | |
| 217 | + User-agent: Timpibot | |
| 218 | + Disallow: / | |
| 219 | + User-agent: Diffbot | |
| 220 | + Disallow: / | |
| 221 | + User-agent: Landau-Media-Spider | |
| 46 | 222 | Disallow: / |
| 47 | 223 | |
| 48 | - User-agent: GPTBot | |
| 224 | + | |
| 225 | + # ========================================== | |
| 226 | + # HISTORISCHE BOTS & MASSEN-DOWNLOADER | |
| 227 | + # ========================================== | |
| 228 | + User-agent: Bloodhound | |
| 229 | + Disallow: / | |
| 230 | + User-agent: cydralspider | |
| 231 | + Disallow: / | |
| 232 | + User-agent: downloadexpress | |
| 233 | + Disallow: / | |
| 234 | + User-agent: gammaSpider | |
| 235 | + Disallow: / | |
| 236 | + User-agent: ObjectsSearch | |
| 237 | + Disallow: / | |
| 238 | + User-agent: Pimptrain | |
| 239 | + Disallow: / | |
| 240 | + User-agent: wapspider | |
| 241 | + Disallow: / | |
| 242 | + User-agent: WebZinger | |
| 243 | + Disallow: / | |
| 244 | + User-agent: Fasterfox | |
| 49 | 245 | Disallow: / |
| 50 | 246 | |
| 51 | - User-agent: Applebot-Extended | |
| 247 | + # ========================================== | |
| 248 | + # SCRAPING FRAMEWORKS & UTILITIES | |
| 249 | + # ========================================== | |
| 250 | + User-agent: Scrapy | |
| 251 | + Disallow: / | |
| 252 | + User-agent: HTTPBannerDetection | |
| 253 | + Disallow: / | |
| 254 | + User-agent: Wget | |
| 52 | 255 | Disallow: / |
| 53 | 256 | |
| 54 | - User-agent: CCBot | |
| 55 | - Disallow: / | |
| 56 | - | |
| 57 | - User-agent: magpie-crawler | |
| 58 | - Disallow: / | |
| 59 | - | |
| 60 | - User-agent: ia_archiver | |
| 61 | - Disallow: / | |
| 62 | - | |
| 63 | - User-Agent: omgili | |
| 64 | - Disallow: / | |
| 65 | - | |
| 66 | - User-Agent: omgilibot | |
| 67 | - Disallow: / | |
| 68 | - | |
| 69 | - User-agent: Baiduspider | |
| 70 | - Disallow: / | |
| 71 | - | |
| 72 | - User-agent: AhrefsBot | |
| 73 | - Disallow: / | |
| 74 | - | |
| 75 | - User-agent: DataForSeoBot | |
| 76 | - Disallow: / | |
| 77 | - | |
| 78 | - User-agent: Yeti | |
| 79 | - Disallow: / | |
| 80 | - | |
| 81 | - User-agent: SemrushBot | |
| 82 | - Disallow: / | |
| 83 | - | |
| 84 | - User-agent: sentibot | |
| 85 | - Disallow: / | |
| 86 | - | |
| 87 | - User-agent: MJ12bot | |
| 88 | - Disallow: / | |
| 89 | - | |
| 90 | - User-agent: Bytespider | |
| 91 | - Disallow: / | |
| 92 | - | |
| 93 | - User-agent: SirdataBot | |
| 94 | - Disallow: / | |
| 95 | - | |
| 96 | - User-agent: LCC | |
| 97 | - Disallow: / | |
| 98 | - | |
| 99 | - User-agent: TurnitinBot | |
| 100 | - Disallow: / | |
| 101 | - | |
| 102 | - User-agent: BLEXBot | |
| 103 | - Disallow: / | |
| 104 | - | |
| 105 | - User-agent: dotbot | |
| 106 | - Disallow: / | |
| 107 | - | |
| 108 | - User-Agent: ImagesiftBot | |
| 109 | - Disallow: / | |
| 110 | - | |
| 111 | - User-agent: anthropic-ai | |
| 112 | - Disallow: / | |
| 113 | - | |
| 114 | - User-agent: Claude-Web | |
| 115 | - Disallow: / | |
| 116 | - | |
| 117 | - User-agent: ClaudeBot | |
| 118 | - Disallow: / | |
| 119 | - | |
| 120 | - User-agent: Timpibot | |
| 121 | - Disallow: / | |
| 122 | - | |
| 123 | - User-agent: cohere-ai | |
| 124 | - Disallow: / | |
| 125 | - | |
| 126 | - User-agent: Meta-ExternalAgent | |
| 127 | - Disallow: / | |
| 128 | - | |
| 129 | - User-agent: FacebookBot | |
| 130 | - Disallow: / | |
| 131 | - | |
| 132 | - User-agent: Diffbot | |
| 133 | - Disallow: / | |
| 134 | - | |
| 257 | + # Sitemaps | |
| 135 | 258 | Sitemap: https://www.spiegel.de/sitemaps/news-de.xml |
| 136 | 259 | Sitemap: https://www.spiegel.de/sitemaps/videos/sitemap.xml |
| 137 | 260 | Sitemap: https://www.spiegel.de/plus/sitemap.xml |