NeuralCrawl

Die Zeit / robots.txt snapshot

← back to zeit.de · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 2134 bytes · sha256 2667046d22466996 · raw

final URL: https://www.zeit.de/robots.txt

1User-agent: Googlebot-News
2Disallow: /angebote/
3
4User-agent: *
5Disallow: /zeit/
6Disallow: /templates/
7Disallow: /hp_channels/
8Disallow: /send/
9Disallow: /rezepte/suche/
10Disallow: */comment-thread?
11Disallow: */liveblog-backend*
12Disallow: /framebuilder/
13Disallow: /campus/framebuilder/
14Disallow: /navigation-teasers*
15Disallow: *iqadcontroller.js
16Allow: /llms.txt
17
18User-agent: anthropic-ai
19Disallow: /
20
21User-agent: Ai2Bot-Dolma
22Disallow: /
23
24User-agent: Applebot-Extended
25Disallow: /
26
27User-agent: Baiduspider
28Disallow: /
29
30User-agent: Bytespider
31Disallow: /
32
33User-agent: CCBot
34Disallow: /
35
36User-agent: ChatGLM-Spider
37Disallow: /
38
39User-agent: ClaudeBot
40Disallow: /
41
42User-agent: CloudVertexBot
43Disallow: /
44
45User-agent: cohere-training-data-crawler
46Disallow: /
47
48User-agent: Cotoyogi
49Disallow: /
50
51User-agent: DeepSeekBot
52Disallow: /
53
54User-agent: Diffbot
55Disallow: /
56
57User-agent: FacebookBot
58Disallow: /
59
60User-agent: Google-CloudVertexBot
61Disallow: /
62
63User-agent: Google-Extended
64Disallow: /
65Allow: /*-gxe$
66
67User-agent: GPTBot
68Disallow: /
69
70User-agent: Google-Extended
71Disallow: /
72Allow: /*-gxe$
73
74User-agent: GrapeshotCrawler
75crawl-delay: 3
76
77User-agent: img2dataset
78Disallow: /
79
80User-agent: Kangaroo Bot
81Disallow: /
82
83User-agent: KunatoCrawler
84Disallow: /
85
86User-agent: Meta-ExternalAgent
87Disallow: /
88
89User-agent: PanguBot
90Disallow: /
91
92User-agent: Perplexity-User
93Disallow: /
94
95User-agent: PerplexityBot
96Disallow: /
97
98User-agent: quillbot.com
99Disallow: /
100
101User-agent: Spider
102Disallow: /
103
104User-agent: TerraCotta
105Disallow: /
106
107User-agent: Timpibot
108Disallow: /
109
110User-agent: VelenPublicWebCrawler
111Disallow: /
112
113
114Sitemap: https://www.zeit.de/gsitemaps/index.xml
115
116# Legal notice: zeit.de expressly reserves the right to use its content for commercial text and data mining (§ 44 b UrhG).
117# The use of robots or other automated means to access zeit.de or collect or mine data without
118# the express permission of zeit.de is strictly prohibited.
119# zeit.de may, in its discretion, permit certain automated access to certain zeit.de pages,
120# If you would like to apply for permission to crawl zeit.de, collect or use data, please email [email protected]