NeuralCrawl

Die Zeit / robots.txt snapshot

← back to zeit.de · fetched 2026-07-02T12:51:16Z (1mo ago) · HTTP 200 · 2152 bytes · sha256 2a11db89face9c40 · raw

final URL: https://www.zeit.de/robots.txt

1User-agent: Googlebot-News
2Disallow: /angebote/
3
4User-agent: *
5Disallow: /zeit/
6Disallow: /suche/
7Disallow: /templates/
8Disallow: /hp_channels/
9Disallow: /send/
10Disallow: /rezepte/suche/
11Disallow: */comment-thread?
12Disallow: */liveblog-backend*
13Disallow: /framebuilder/
14Disallow: /campus/framebuilder/
15Disallow: /navigation-teasers*
16Disallow: *iqadcontroller.js
17Allow: /llms.txt
18
19User-agent: anthropic-ai
20Disallow: /
21
22User-agent: Ai2Bot-Dolma
23Disallow: /
24
25User-agent: Applebot-Extended
26Disallow: /
27
28User-agent: Baiduspider
29Disallow: /
30
31User-agent: Bytespider
32Disallow: /
33
34User-agent: CCBot
35Disallow: /
36
37User-agent: ChatGLM-Spider
38Disallow: /
39
40User-agent: ClaudeBot
41Disallow: /
42
43User-agent: CloudVertexBot
44Disallow: /
45
46User-agent: cohere-training-data-crawler
47Disallow: /
48
49User-agent: Cotoyogi
50Disallow: /
51
52User-agent: DeepSeekBot
53Disallow: /
54
55User-agent: Diffbot
56Disallow: /
57
58User-agent: FacebookBot
59Disallow: /
60
61User-agent: Google-CloudVertexBot
62Disallow: /
63
64User-agent: Google-Extended
65Disallow: /
66Allow: /*-gxe$
67
68User-agent: GPTBot
69Disallow: /
70
71User-agent: Google-Extended
72Disallow: /
73Allow: /*-gxe$
74
75User-agent: GrapeshotCrawler
76crawl-delay: 3
77
78User-agent: img2dataset
79Disallow: /
80
81User-agent: Kangaroo Bot
82Disallow: /
83
84User-agent: KunatoCrawler
85Disallow: /
86
87User-agent: Meta-ExternalAgent
88Disallow: /
89
90User-agent: PanguBot
91Disallow: /
92
93User-agent: Perplexity-User
94Disallow: /
95
96User-agent: PerplexityBot
97Disallow: /
98
99User-agent: quillbot.com
100Disallow: /
101
102User-agent: Spider
103Disallow: /
104
105User-agent: TerraCotta
106Disallow: /
107
108User-agent: Timpibot
109Disallow: /
110
111User-agent: VelenPublicWebCrawler
112Disallow: /
113
114
115Sitemap: https://www.zeit.de/gsitemaps/index.xml
116
117# Legal notice: zeit.de expressly reserves the right to use its content for commercial text and data mining (§ 44 b UrhG).
118# The use of robots or other automated means to access zeit.de or collect or mine data without
119# the express permission of zeit.de is strictly prohibited.
120# zeit.de may, in its discretion, permit certain automated access to certain zeit.de pages,
121# If you would like to apply for permission to crawl zeit.de, collect or use data, please email [email protected]