NeuralCrawl

Der Spiegel / robots.txt snapshot

← back to spiegel.de · fetched 2026-09-29T12:52:06Z (1mo ago) · HTTP 200 · 6667 bytes · sha256 891e741cb287eda0 · raw

final URL: https://www.spiegel.de/robots.txt

1User-agent: *
2Allow: /
3Disallow: /*CR-Dokumentation.pdf$
4
5User-agent: emetriqContextualBot
6Allow: /
7
8User-agent: Mozilla/5.0 (compatible; OGDWCtxCrawler)
9Allow: /
10
11User-agent: AmazonAdBot
12Allow: /
13
14# TLP-6507: Testweise Freischaltung der OpenAI-Suchcrawler fuer ausgewaehlte Bereiche
15User-agent: OAI-SearchBot
16Allow: /ausland/
17Allow: /partnerschaft/
18Allow: /gesundheit/
19Allow: /familie/
20Allow: /reise/
21Allow: /psychologie/
22Allow: /stil/
23Allow: /tests/
24Disallow: /
25
26# TLP-6507: Testweise Freischaltung der OpenAI-Suchcrawler fuer ausgewaehlte Bereiche
27User-agent: ChatGPT-User
28Allow: /ausland/
29Allow: /partnerschaft/
30Allow: /gesundheit/
31Allow: /familie/
32Allow: /reise/
33Allow: /psychologie/
34Allow: /stil/
35Allow: /tests/
36Disallow: /
37
38# TLP-7042: Testweise Freischaltung Claude-Web fuer ausgewaehlte Bereiche
39User-agent: Claude-Web
40Allow: /ausland/
41Allow: /partnerschaft/
42Allow: /gesundheit/
43Allow: /familie/
44Allow: /reise/
45Allow: /psychologie/
46Allow: /stil/
47Allow: /tests/
48Disallow: /
49
50# TLP-7042: Testweise Freischaltung Claude-SearchBot fuer ausgewaehlte Bereiche
51User-agent: Claude-SearchBot
52Allow: /ausland/
53Allow: /partnerschaft/
54Allow: /gesundheit/
55Allow: /familie/
56Allow: /reise/
57Allow: /psychologie/
58Allow: /stil/
59Allow: /tests/
60Disallow: /
61
62# TLP-7042: Testweise Freischaltung Claude-User fuer ausgewaehlte Bereiche
63User-agent: Claude-User
64Allow: /ausland/
65Allow: /partnerschaft/
66Allow: /gesundheit/
67Allow: /familie/
68Allow: /reise/
69Allow: /psychologie/
70Allow: /stil/
71Allow: /tests/
72Disallow: /
73
74# ==========================================
75# KI-CRAWLER & KI-TRAINING (SPERRE)
76# ==========================================
77User-agent: GPTBot
78Disallow: /
79User-agent: anthropic-ai
80Disallow: /
81User-agent: ClaudeBot
82Disallow: /
83User-agent: CloudVertexBot
84Disallow: /
85User-agent: cohere-ai
86Disallow: /
87User-agent: cohere-training-data-crawler
88Disallow: /
89User-agent: cohere-training-data-collector
90Disallow: /
91User-agent: DeepSeekBot
92Disallow: /
93User-agent: DeepSeek
94Disallow: /
95User-agent: Meta-ExternalAgent
96Disallow: /
97User-agent: FacebookBot
98Disallow: /
99User-agent: Applebot-Extended
100Disallow: /
101User-agent: MistralAI-User
102Disallow: /
103User-agent: Amazonbot
104Disallow: /
105User-agent: AI2Bot
106Disallow: /
107User-agent: Kangaroo Bot
108Disallow: /
109User-agent: PanguBot
110Disallow: /
111User-agent: PetalBot
112Disallow: /
113User-agent: Devin
114Disallow: /
115User-agent: bigsur.ai
116Disallow: /
117User-agent: LinerBot
118Disallow: /
119User-agent: CCBot
120Disallow: /
121User-agent: YouBot
122Disallow: /
123User-agent: YouBot-Search
124Disallow: /
125User-agent: iaskspider
126Disallow: /
127User-agent: YoudaoBot
128Disallow: /
129User-agent: DeepL-Translate
130Disallow: /
131User-agent: DeepL-Bot
132Disallow: /
133
134# ==========================================
135# SEO-CRAWLER (SPERRE)
136# ==========================================
137User-agent: AhrefsBot
138Disallow: /
139User-agent: AhrefsSiteAudit
140Disallow: /
141User-agent: SemrushBot
142Disallow: /
143User-agent: SemrushBot-SA
144Disallow: /
145User-agent: SemrushBot-BA
146Disallow: /
147User-agent: SemrushBot-SI
148Disallow: /
149User-agent: SemrushBot-SWA
150Disallow: /
151User-agent: SiteAuditBot
152Disallow: /
153User-agent: SplitSignalBot
154Disallow: /
155User-agent: xovi
156Disallow: /
157User-agent: XoviBot
158Disallow: /
159User-agent: Seobility
160Disallow: /
161User-agent: SeobilityBot
162Disallow: /
163User-agent: RyteBot
164Disallow: /
165User-agent: SEOkicks
166Disallow: /
167User-agent: SEOkicks-Robot
168Disallow: /
169User-agent: Searchmetrics
170Disallow: /
171User-agent: SearchmetricsBot
172Disallow: /
173User-agent: audisto
174Disallow: /
175User-agent: audisto-essential
176Disallow: /
177User-agent: MJ12bot
178Disallow: /
179User-agent: dotbot
180Disallow: /
181User-agent: rogerbot
182Disallow: /
183User-agent: Ezooms
184Disallow: /
185User-agent: barkrowler
186Disallow: /
187User-agent: serpstatbot
188Disallow: /
189User-agent: BLEXBot
190Disallow: /
191User-agent: DataForSeoBot
192Disallow: /
193User-agent: MegaIndex
194Disallow: /
195User-agent: lumar
196Disallow: /
197User-agent: deepcrawl
198Disallow: /
199User-agent: oncrawl
200Disallow: /
201User-agent: sitebulb
202Disallow: /
203User-agent: botify
204Disallow: /
205User-agent: Siteimprove
206Disallow: /
207User-agent: Siteimprove Crawl
208Disallow: /
209User-agent: MozDotNet
210Disallow: /
211User-agent: Cocolyzebot
212Disallow: /
213User-agent: Raven
214Disallow: /
215
216# ==========================================
217# MEDIENBEOBACHTUNG, AGGREGATOREN & SCRAPER
218# ==========================================
219User-agent: Meltwater
220Disallow: /
221User-agent: NewsNow
222Disallow: /
223User-agent: Webzio-Extended
224Disallow: /
225User-agent: magpie-crawler
226Disallow: /
227User-agent: omgili
228Disallow: /
229User-agent: omgilibot
230Disallow: /
231User-agent: Baiduspider
232Disallow: /
233User-agent: Yeti
234Disallow: /
235User-agent: sentibot
236Disallow: /
237User-agent: Bytespider
238Disallow: /
239User-agent: SirdataBot
240Disallow: /
241User-agent: LCC
242Disallow: /
243User-agent: TurnitinBot
244Disallow: /
245User-agent: ImagesiftBot
246Disallow: /
247User-agent: Timpibot
248Disallow: /
249User-agent: Diffbot
250Disallow: /
251User-agent: Landau-Media-Spider
252Disallow: /
253
254
255# ==========================================
256# ARCHIV-CRAWLER (SPERRE)
257# ==========================================
258User-agent: archive.org_bot
259Disallow: /
260User-agent: WaybackSavePage
261Disallow: /
262User-agent: heritrix
263Disallow: /
264User-agent: Special_Archiver
265Disallow: /
266User-agent: ia_archiver-web.archive.org
267Disallow: /
268User-agent: ia_archiver2
269Disallow: /
270User-agent: ia_archiver
271Disallow: /
272
273# ==========================================
274# HISTORISCHE BOTS & MASSEN-DOWNLOADER
275# ==========================================
276User-agent: Bloodhound
277Disallow: /
278User-agent: cydralspider
279Disallow: /
280User-agent: downloadexpress
281Disallow: /
282User-agent: gammaSpider
283Disallow: /
284User-agent: ObjectsSearch
285Disallow: /
286User-agent: Pimptrain
287Disallow: /
288User-agent: wapspider
289Disallow: /
290User-agent: WebZinger
291Disallow: /
292User-agent: Fasterfox
293Disallow: /
294
295# ==========================================
296# SCRAPING FRAMEWORKS & UTILITIES
297# ==========================================
298User-agent: Scrapy
299Disallow: /
300User-agent: HTTPBannerDetection
301Disallow: /
302User-agent: Wget
303Disallow: /
304
305# Sitemaps
306Sitemap: https://www.spiegel.de/sitemaps/news-de.xml
307Sitemap: https://www.spiegel.de/sitemaps/videos/sitemap.xml
308Sitemap: https://www.spiegel.de/plus/sitemap.xml
309Sitemap: https://www.spiegel.de/sitemap.xml
310
311# Legal notice: spiegel.de expressly reserves the right to use its content for commercial text and data mining (§ 44b Urheberrechtsgesetz).
312# The use of robots or other automated means to access spiegel.de or collect or mine data without the express permission of spiegel.de is strictly prohibited.
313# spiegel.de may, in its discretion, permit certain automated access to certain spiegel.de pages,
314# If you would like to apply for permission to crawl spiegel.de, collect or use data, please email [email protected]
315