NeuralCrawl

Der Spiegel / robots.txt snapshot

← back to spiegel.de · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 2617 bytes · sha256 2bb6d059aa71e0a8 · raw

final URL: https://www.spiegel.de/robots.txt

1User-agent: *
2Allow: /
3Disallow: /*CR-Dokumentation.pdf$
4
5User-agent: emetriqContextualBot
6Allow: /
7
8User-agent: Mozilla/5.0 (compatible; OGDWCtxCrawler)
9Allow: /
10
11User-agent: AmazonAdBot
12Allow: /
13
14# TLP-6507: Testweise Freischaltung der OpenAI-Suchcrawler fuer ausgewaehlte Bereiche
15User-agent: OAI-SearchBot
16Allow: /ausland/
17Allow: /partnerschaft/
18Allow: /gesundheit/
19Allow: /familie/
20Allow: /reise/
21Allow: /psychologie/
22Allow: /stil/
23Disallow: /
24
25# TLP-6507: Testweise Freischaltung der OpenAI-Suchcrawler fuer ausgewaehlte Bereiche
26User-agent: ChatGPT-User
27Allow: /ausland/
28Allow: /partnerschaft/
29Allow: /gesundheit/
30Allow: /familie/
31Allow: /reise/
32Allow: /psychologie/
33Allow: /stil/
34Disallow: /
35
36User-agent: cohere-ai
37Disallow: /
38
39User-agent: cohere-training-data-crawler
40Disallow: /
41
42User-agent: Webzio-Extended
43Disallow: /
44
45User-agent: YouBot
46Disallow: /
47
48User-agent: GPTBot
49Disallow: /
50
51User-agent: Applebot-Extended
52Disallow: /
53
54User-agent: CCBot
55Disallow: /
56
57User-agent: magpie-crawler
58Disallow: /
59
60User-agent: ia_archiver
61Disallow: /
62
63User-Agent: omgili
64Disallow: /
65
66User-Agent: omgilibot
67Disallow: /
68
69User-agent: Baiduspider
70Disallow: /
71
72User-agent: AhrefsBot
73Disallow: /
74
75User-agent: DataForSeoBot
76Disallow: /
77
78User-agent: Yeti
79Disallow: /
80
81User-agent: SemrushBot
82Disallow: /
83
84User-agent: sentibot
85Disallow: /
86
87User-agent: MJ12bot
88Disallow: /
89
90User-agent: Bytespider
91Disallow: /
92
93User-agent: SirdataBot
94Disallow: /
95
96User-agent: LCC
97Disallow: /
98
99User-agent: TurnitinBot
100Disallow: /
101
102User-agent: BLEXBot
103Disallow: /
104
105User-agent: dotbot
106Disallow: /
107
108User-Agent: ImagesiftBot
109Disallow: /
110
111User-agent: anthropic-ai
112Disallow: /
113
114User-agent: Claude-Web
115Disallow: /
116
117User-agent: ClaudeBot
118Disallow: /
119
120User-agent: Timpibot
121Disallow: /
122
123User-agent: cohere-ai
124Disallow: /
125
126User-agent: Meta-ExternalAgent
127Disallow: /
128
129User-agent: FacebookBot
130Disallow: /
131
132User-agent: Diffbot
133Disallow: /
134
135Sitemap: https://www.spiegel.de/sitemaps/news-de.xml
136Sitemap: https://www.spiegel.de/sitemaps/videos/sitemap.xml
137Sitemap: https://www.spiegel.de/plus/sitemap.xml
138Sitemap: https://www.spiegel.de/sitemap.xml
139
140# Legal notice: spiegel.de expressly reserves the right to use its content for commercial text and data mining (§ 44b Urheberrechtsgesetz).
141# The use of robots or other automated means to access spiegel.de or collect or mine data without the express permission of spiegel.de is strictly prohibited.
142# spiegel.de may, in its discretion, permit certain automated access to certain spiegel.de pages,
143# If you would like to apply for permission to crawl spiegel.de, collect or use data, please email [email protected]
144