NeuralCrawl

Der Spiegel / robots.txt snapshot

← back to spiegel.de · fetched 2026-10-13T11:46:10Z (24d ago) · HTTP 200 · 6633 bytes · sha256 95f0a83e7d047a5a · raw

final URL: https://www.spiegel.de/robots.txt

1User-agent: *
2Allow: /
3Disallow: /*CR-Dokumentation.pdf$
4
5User-agent: emetriqContextualBot
6Allow: /
7
8User-agent: Mozilla/5.0 (compatible; OGDWCtxCrawler)
9Allow: /
10
11User-agent: AmazonAdBot
12Allow: /
13
14# TLP-6507: Testweise Freischaltung der OpenAI-Suchcrawler fuer ausgewaehlte Bereiche
15User-agent: OAI-SearchBot
16Allow: /ausland/
17Allow: /partnerschaft/
18Allow: /gesundheit/
19Allow: /familie/
20Allow: /reise/
21Allow: /psychologie/
22Allow: /stil/
23Allow: /tests/
24Disallow: /
25
26# TLP-6507: Testweise Freischaltung der OpenAI-Suchcrawler fuer ausgewaehlte Bereiche
27User-agent: ChatGPT-User
28Allow: /ausland/
29Allow: /partnerschaft/
30Allow: /gesundheit/
31Allow: /familie/
32Allow: /reise/
33Allow: /psychologie/
34Allow: /stil/
35Allow: /tests/
36Disallow: /
37
38# TLP-7042: Testweise Freischaltung Claude-Web fuer ausgewaehlte Bereiche
39User-agent: Claude-Web
40Allow: /ausland/
41Allow: /partnerschaft/
42Allow: /gesundheit/
43Allow: /familie/
44Allow: /reise/
45Allow: /psychologie/
46Allow: /stil/
47Allow: /tests/
48Disallow: /
49
50# TLP-7042: Testweise Freischaltung Claude-SearchBot fuer ausgewaehlte Bereiche
51User-agent: Claude-SearchBot
52Allow: /ausland/
53Allow: /partnerschaft/
54Allow: /gesundheit/
55Allow: /familie/
56Allow: /reise/
57Allow: /psychologie/
58Allow: /stil/
59Allow: /tests/
60Disallow: /
61
62# TLP-7042: Testweise Freischaltung Claude-User fuer ausgewaehlte Bereiche
63User-agent: Claude-User
64Allow: /ausland/
65Allow: /partnerschaft/
66Allow: /gesundheit/
67Allow: /familie/
68Allow: /reise/
69Allow: /psychologie/
70Allow: /stil/
71Allow: /tests/
72Disallow: /
73
74# ==========================================
75# KI-CRAWLER & KI-TRAINING (SPERRE)
76# ==========================================
77User-agent: GPTBot
78Disallow: /
79User-agent: anthropic-ai
80Disallow: /
81User-agent: ClaudeBot
82Disallow: /
83User-agent: CloudVertexBot
84Disallow: /
85User-agent: cohere-ai
86Disallow: /
87User-agent: cohere-training-data-crawler
88Disallow: /
89User-agent: cohere-training-data-collector
90Disallow: /
91User-agent: DeepSeekBot
92Disallow: /
93User-agent: DeepSeek
94Disallow: /
95User-agent: Meta-ExternalAgent
96Disallow: /
97User-agent: FacebookBot
98Disallow: /
99User-agent: Applebot-Extended
100Disallow: /
101User-agent: MistralAI-User
102Disallow: /
103User-agent: AI2Bot
104Disallow: /
105User-agent: Kangaroo Bot
106Disallow: /
107User-agent: PanguBot
108Disallow: /
109User-agent: PetalBot
110Disallow: /
111User-agent: Devin
112Disallow: /
113User-agent: bigsur.ai
114Disallow: /
115User-agent: LinerBot
116Disallow: /
117User-agent: CCBot
118Disallow: /
119User-agent: YouBot
120Disallow: /
121User-agent: YouBot-Search
122Disallow: /
123User-agent: iaskspider
124Disallow: /
125User-agent: YoudaoBot
126Disallow: /
127User-agent: DeepL-Translate
128Disallow: /
129User-agent: DeepL-Bot
130Disallow: /
131
132# ==========================================
133# SEO-CRAWLER (SPERRE)
134# ==========================================
135User-agent: AhrefsBot
136Disallow: /
137User-agent: AhrefsSiteAudit
138Disallow: /
139User-agent: SemrushBot
140Disallow: /
141User-agent: SemrushBot-SA
142Disallow: /
143User-agent: SemrushBot-BA
144Disallow: /
145User-agent: SemrushBot-SI
146Disallow: /
147User-agent: SemrushBot-SWA
148Disallow: /
149User-agent: SiteAuditBot
150Disallow: /
151User-agent: SplitSignalBot
152Disallow: /
153User-agent: xovi
154Disallow: /
155User-agent: XoviBot
156Disallow: /
157User-agent: Seobility
158Disallow: /
159User-agent: SeobilityBot
160Disallow: /
161User-agent: RyteBot
162Disallow: /
163User-agent: SEOkicks
164Disallow: /
165User-agent: SEOkicks-Robot
166Disallow: /
167User-agent: Searchmetrics
168Disallow: /
169User-agent: SearchmetricsBot
170Disallow: /
171User-agent: audisto
172Disallow: /
173User-agent: audisto-essential
174Disallow: /
175User-agent: MJ12bot
176Disallow: /
177User-agent: dotbot
178Disallow: /
179User-agent: rogerbot
180Disallow: /
181User-agent: Ezooms
182Disallow: /
183User-agent: barkrowler
184Disallow: /
185User-agent: serpstatbot
186Disallow: /
187User-agent: BLEXBot
188Disallow: /
189User-agent: DataForSeoBot
190Disallow: /
191User-agent: MegaIndex
192Disallow: /
193User-agent: lumar
194Disallow: /
195User-agent: deepcrawl
196Disallow: /
197User-agent: oncrawl
198Disallow: /
199User-agent: sitebulb
200Disallow: /
201User-agent: botify
202Disallow: /
203User-agent: Siteimprove
204Disallow: /
205User-agent: Siteimprove Crawl
206Disallow: /
207User-agent: MozDotNet
208Disallow: /
209User-agent: Cocolyzebot
210Disallow: /
211User-agent: Raven
212Disallow: /
213
214# ==========================================
215# MEDIENBEOBACHTUNG, AGGREGATOREN & SCRAPER
216# ==========================================
217User-agent: Meltwater
218Disallow: /
219User-agent: NewsNow
220Disallow: /
221User-agent: Webzio-Extended
222Disallow: /
223User-agent: magpie-crawler
224Disallow: /
225User-agent: omgili
226Disallow: /
227User-agent: omgilibot
228Disallow: /
229User-agent: Baiduspider
230Disallow: /
231User-agent: Yeti
232Disallow: /
233User-agent: sentibot
234Disallow: /
235User-agent: Bytespider
236Disallow: /
237User-agent: SirdataBot
238Disallow: /
239User-agent: LCC
240Disallow: /
241User-agent: TurnitinBot
242Disallow: /
243User-agent: ImagesiftBot
244Disallow: /
245User-agent: Timpibot
246Disallow: /
247User-agent: Diffbot
248Disallow: /
249User-agent: Landau-Media-Spider
250Disallow: /
251
252
253# ==========================================
254# ARCHIV-CRAWLER (SPERRE)
255# ==========================================
256User-agent: archive.org_bot
257Disallow: /
258User-agent: WaybackSavePage
259Disallow: /
260User-agent: heritrix
261Disallow: /
262User-agent: Special_Archiver
263Disallow: /
264User-agent: ia_archiver-web.archive.org
265Disallow: /
266User-agent: ia_archiver2
267Disallow: /
268User-agent: ia_archiver
269Disallow: /
270
271# ==========================================
272# HISTORISCHE BOTS & MASSEN-DOWNLOADER
273# ==========================================
274User-agent: Bloodhound
275Disallow: /
276User-agent: cydralspider
277Disallow: /
278User-agent: downloadexpress
279Disallow: /
280User-agent: gammaSpider
281Disallow: /
282User-agent: ObjectsSearch
283Disallow: /
284User-agent: Pimptrain
285Disallow: /
286User-agent: wapspider
287Disallow: /
288User-agent: WebZinger
289Disallow: /
290User-agent: Fasterfox
291Disallow: /
292
293# ==========================================
294# SCRAPING FRAMEWORKS & UTILITIES
295# ==========================================
296User-agent: Scrapy
297Disallow: /
298User-agent: HTTPBannerDetection
299Disallow: /
300User-agent: Wget
301Disallow: /
302
303# Sitemaps
304Sitemap: https://www.spiegel.de/sitemaps/news-de.xml
305Sitemap: https://www.spiegel.de/sitemaps/videos/sitemap.xml
306Sitemap: https://www.spiegel.de/plus/sitemap.xml
307Sitemap: https://www.spiegel.de/sitemap.xml
308
309# Legal notice: spiegel.de expressly reserves the right to use its content for commercial text and data mining (§ 44b Urheberrechtsgesetz).
310# The use of robots or other automated means to access spiegel.de or collect or mine data without the express permission of spiegel.de is strictly prohibited.
311# spiegel.de may, in its discretion, permit certain automated access to certain spiegel.de pages,
312# If you would like to apply for permission to crawl spiegel.de, collect or use data, please email [email protected]
313