NeuralCrawl

Die Zeit / robots.txt snapshot

← back to zeit.de · fetched 2026-08-31T15:31:09Z (2mo ago) · HTTP 200 · 2244 bytes · sha256 cb57f6490222e9af · raw

final URL: https://www.zeit.de/robots.txt

1# FALLBACK ONLY, instead edit here: https://dash.centinelanalytica.com/crawlers/robots-txt
2
3User-agent: Googlebot-News
4Disallow: /angebote/
5
6User-agent: *
7Disallow: /zeit/
8Disallow: /suche/
9Disallow: /templates/
10Disallow: /hp_channels/
11Disallow: /send/
12Disallow: /rezepte/suche/
13Disallow: */comment-thread?
14Disallow: */liveblog-backend*
15Disallow: /framebuilder/
16Disallow: /campus/framebuilder/
17Disallow: /navigation-teasers*
18Disallow: *iqadcontroller.js
19Allow: /llms.txt
20
21User-agent: anthropic-ai
22Disallow: /
23
24User-agent: Ai2Bot-Dolma
25Disallow: /
26
27User-agent: Applebot-Extended
28Disallow: /
29
30User-agent: Baiduspider
31Disallow: /
32
33User-agent: Bytespider
34Disallow: /
35
36User-agent: CCBot
37Disallow: /
38
39User-agent: ChatGLM-Spider
40Disallow: /
41
42User-agent: ClaudeBot
43Disallow: /
44
45User-agent: CloudVertexBot
46Disallow: /
47
48User-agent: cohere-training-data-crawler
49Disallow: /
50
51User-agent: Cotoyogi
52Disallow: /
53
54User-agent: DeepSeekBot
55Disallow: /
56
57User-agent: Diffbot
58Disallow: /
59
60User-agent: FacebookBot
61Disallow: /
62
63User-agent: Google-CloudVertexBot
64Disallow: /
65
66User-agent: Google-Extended
67Disallow: /
68Allow: /*-gxe$
69
70User-agent: GPTBot
71Disallow: /
72
73User-agent: Google-Extended
74Disallow: /
75Allow: /*-gxe$
76
77User-agent: GrapeshotCrawler
78crawl-delay: 3
79
80User-agent: img2dataset
81Disallow: /
82
83User-agent: Kangaroo Bot
84Disallow: /
85
86User-agent: KunatoCrawler
87Disallow: /
88
89User-agent: Meta-ExternalAgent
90Disallow: /
91
92User-agent: PanguBot
93Disallow: /
94
95User-agent: Perplexity-User
96Disallow: /
97
98User-agent: PerplexityBot
99Disallow: /
100
101User-agent: quillbot.com
102Disallow: /
103
104User-agent: Spider
105Disallow: /
106
107User-agent: TerraCotta
108Disallow: /
109
110User-agent: Timpibot
111Disallow: /
112
113User-agent: VelenPublicWebCrawler
114Disallow: /
115
116
117Sitemap: https://www.zeit.de/gsitemaps/index.xml
118
119# Legal notice: zeit.de expressly reserves the right to use its content for commercial text and data mining (§ 44 b UrhG).
120# The use of robots or other automated means to access zeit.de or collect or mine data without
121# the express permission of zeit.de is strictly prohibited.
122# zeit.de may, in its discretion, permit certain automated access to certain zeit.de pages,
123# If you would like to apply for permission to crawl zeit.de, collect or use data, please email [email protected]