NeuralCrawl

Die Zeit / robots.txt

← back to zeit.de · change detected 9h ago (2026-11-09T15:07:03Z) · modified +354 −94

What changed

Diff old (2026-09-11T13:10:12Z) → new (2026-11-09T15:07:03Z)

@@ -1,7 +1,346 @@
1+ User-agent: AASA-Bot
2+ User-agent: Accessible Web Bot
3+ User-agent: AccessibleWebBot
4+ User-agent: AdDefend
5+ User-agent: adidxbot
6+ User-agent: AdkernelTopicCrawler
7+ User-agent: AdsBot-Google
8+ User-agent: AdsBot-Google-Mobile
9+ User-agent: Adstxt Parser, Demandbase
10+ User-agent: AdsTxtCrawler
11+ User-agent: Akkoma
12+ User-agent: Amazon AdBot
13+ User-agent: Amazon-Advertising-ad-standards-bot
14+ User-agent: AmazonAdBot
15+ User-agent: APIs-Google
16+ User-agent: Apple AuthenticationServicesAgent
17+ User-agent: Apple Link Preview
18+ User-agent: Applebot
19+ User-agent: AppleNewsBot
20+ User-agent: ArachnysSpider
21+ User-agent: Assaybot
22+ User-agent: Automattic
23+ User-agent: Awario
24+ User-agent: AwarioBot
25+ User-agent: AwarioRssBot
26+ User-agent: AwarioSmartBot
27+ User-agent: BazQux
28+ User-agent: Bing Ads
29+ User-agent: Bing Preview
30+ User-agent: bingbot
31+ User-agent: bingbot/
32+ User-agent: BingPreview
33+ User-agent: bitlybot
34+ User-agent: Bluesky
35+ User-agent: botify
36+ User-agent: BrandVerity
37+ User-agent: brokenlinkcheck
38+ User-agent: Brusselsbot
39+ User-agent: Buck/
40+ User-agent: Buckbot
41+ User-agent: BufferLinkPreviewBot
42+ User-agent: captify-crawler
43+ User-agent: Chrome Privacy Preserving Prefetch Proxy
44+ User-agent: Chrome-Lighthouse
45+ User-agent: citoid
46+ User-agent: Citoid/WMF
47+ User-agent: CommaFeed
48+ User-agent: compatible; proximic
49+ User-agent: Country Feed Monitor
50+ User-agent: CriteoBot
51+ User-agent: CriteoBot/
52+ User-agent: Cxense
53+ User-agent: cXensebot
54+ User-agent: cXensebot/
55+ User-agent: deadlinkchecker
56+ User-agent: Discordbot
57+ User-agent: dlvr.it
58+ User-agent: dlvrit
59+ User-agent: DobbyNewsBot
60+ User-agent: Domains
61+ User-agent: domains-distillery
62+ User-agent: DoubleVerifyBot
63+ User-agent: DuckDuckBot
64+ User-agent: DVbot
65+ User-agent: Echobot Bot
66+ User-agent: Edge Health Probe
67+ User-agent: ElevenlabsBot
68+ User-agent: ElevenlabsCrawler
69+ User-agent: Embedly
70+ User-agent: emetriqContextualBot
71+ User-agent: EventRadar/1.0
72+ User-agent: ExaUser
73+ User-agent: ExteAdstxtCrawl
74+ User-agent: facebookcatalog
75+ User-agent: facebookexternalhit
76+ User-agent: facebookexternalua
77+ User-agent: facebookplatform
78+ User-agent: Facebot
79+ User-agent: FairOpenNewsBot
80+ User-agent: FAST-WebCrawler
81+ User-agent: Fastmail Bot
82+ User-agent: FastmailUA
83+ User-agent: Feedbin
84+ User-agent: feedbot
85+ User-agent: FeedBurner
86+ User-agent: Feeder /
87+ User-agent: feeder.co
88+ User-agent: FeedFetcher-Google
89+ User-agent: FeedFlow
90+ User-agent: Feedler/1.0
91+ User-agent: Feedly
92+ User-agent: Feedly/1
93+ User-agent: Feedsearch
94+ User-agent: Feedsearch-Crawler
95+ User-agent: FeedValidator
96+ User-agent: FleebsBot
97+ User-agent: FlipboardProxy
98+ User-agent: FlipboardRSS
99+ User-agent: FreshRSS
100+ User-agent: Friendica
101+ User-agent: Gettrbot
102+ User-agent: GlancelotBot
103+ User-agent: GnowitNewsbot
104+ User-agent: GoodNews-Crawler
105+ User-agent: Google AdsBot
106+ User-agent: Google AdSense
107+ User-agent: Google AppsViewer
108+ User-agent: Google Digital Asset Links
109+ User-agent: Google Favicon
110+ User-agent: Google Feed Fetcher
111+ User-agent: Google Image Proxy
112+ User-agent: Google Images
113+ User-agent: Google Inspection Tool
114+ User-agent: Google Messages Link Preview
115+ User-agent: Google Publisher Center
116+ User-agent: Google Schema Markup Testing Tool
117+ User-agent: Google Scholar
118+ User-agent: Google StoreBot
119+ User-agent: Google Videos
120+ User-agent: Google Web Preview
121+ User-agent: Google-Ads-Creatives-Assistant
122+ User-agent: Google-adstxt
123+ User-agent: Google-AdWords-Express
124+ User-agent: Google-Adwords-Instant
125+ User-agent: Google-BusinessLinkVerification
126+ User-agent: Google-Calendar-Importer
127+ User-agent: Google-Certificates-Bridge
128+ User-agent: Google-Display-Ads-Bot
129+ User-agent: Google-Flights-Search-Spackle
130+ User-agent: Google-InspectionTool
131+ User-agent: Google-NotebookLM
132+ User-agent: Google-PageRenderer
133+ User-agent: Google-PhysicalWeb
134+ User-agent: Google-Read-Aloud
135+ User-agent: Google-Safety
136+ User-agent: Google-Site-Verification
137+ User-agent: Google-Sitemaps
138+ User-agent: Google-Structured-Data-Testing
139+ User-agent: Google-Structured-Data-Testing-Tool
140+ User-agent: Google-TravelAds-Live
141+ User-agent: Google-Trust-Services
142+ User-agent: google-xrawler
143+ User-agent: google.com/feedfetcher
144+ User-agent: Google_InspectionTool
145+ User-agent: GoogleAssociationService
146+ User-agent: Googlebot
147+ User-agent: Googlebot-IA
148+ User-agent: Googlebot-IA/
149+ User-agent: Googlebot-Image
150+ User-agent: Googlebot-Mobile
1151  User-agent: Googlebot-News
2- Disallow: /angebote/
152+ User-agent: Googlebot-Video
153+ User-agent: GoogleDocs
154+ User-agent: GoogleImageProxy
155+ User-agent: GoogleOther
156+ User-agent: GoogleOther-Image
157+ User-agent: GoogleOther-Video
158+ User-agent: GoogleProducer
159+ User-agent: GoogleStackdriverMonitoring-UptimeChecks
160+ User-agent: GoogleUserTriggeredFetcher
161+ User-agent: GrapeshotCrawler
162+ User-agent: Greppr
163+ User-agent: GumGum
164+ User-agent: Hootsuite-Authoring
165+ User-agent: HubSpot
166+ User-agent: HubSpot Crawler
167+ User-agent: IAB-Tech-Lab
168+ User-agent: ias-ie
169+ User-agent: ias-or
170+ User-agent: ias-va
171+ User-agent: ias_crawler
172+ User-agent: Iframely
173+ User-agent: IFTTT
174+ User-agent: Inoreader
175+ User-agent: inoreader.com
176+ User-agent: InteractiveAdvertisingBureau
177+ User-agent: IServ News Bot
178+ User-agent: Jetslide
179+ User-agent: JournalistAppBot
180+ User-agent: Leikibot
181+ User-agent: LetsEncrypt
182+ User-agent: LinkedInBot
183+ User-agent: Magus
184+ User-agent: Marginalia Search
185+ User-agent: Mastodon
186+ User-agent: Mediapartners-Google
187+ User-agent: meta-externalads
188+ User-agent: MetaInspector
189+ User-agent: MetaJobBot
190+ User-agent: Microsoft News Ingestion
191+ User-agent: Microsoft Preview
192+ User-agent: MicrosoftPreview
193+ User-agent: MicrosoftPreview/
194+ User-agent: Miniflux
195+ User-agent: MJ12bot
196+ User-agent: MochiImageCrawler
197+ User-agent: Monsidobot
198+ User-agent: MS Search 4.0 Robot
199+ User-agent: MS Search 6.0 Robot
200+ User-agent: MSMNewsBot
201+ User-agent: msnbot
202+ User-agent: msnbot-media
203+ User-agent: MuckRack
204+ User-agent: NanoInteractive
205+ User-agent: NetNewsWire
206+ User-agent: Netvibes
207+ User-agent: NewsBlur
208+ User-agent: NewsCrawler
209+ User-agent: Newslitbot
210+ User-agent: NewsNow
211+ User-agent: Newstral
212+ User-agent: NewsVergleichDE
213+ User-agent: NewsWallBot
214+ User-agent: NewzBot
215+ User-agent: NextCloud
216+ User-agent: Nextcloud Server Crawler
217+ User-agent: Nextcloud-Server-Crawler
218+ User-agent: NoahBot
219+ User-agent: Nooshub
220+ User-agent: OdklBot/
221+ User-agent: officestorebot
222+ User-agent: OpenGraph.io
223+ User-agent: OpenGraphBot
224+ User-agent: OpenindexSpider
225+ User-agent: outbrain
226+ User-agent: Overcast
227+ User-agent: Overcast/1.0 Podcast Sync
228+ User-agent: Ozon Web Grabber
229+ User-agent: Paqlebot
230+ User-agent: peer39_crawler
231+ User-agent: peer39_crawler/1.0
232+ User-agent: PetalBot
233+ User-agent: Pingdom
234+ User-agent: Pinterest
235+ User-agent: pinterest.com/
236+ User-agent: Pinterestbot
237+ User-agent: Pleroma
238+ User-agent: Pocket Casts Feed Parser
239+ User-agent: PocketImageCache
240+ User-agent: Podcatcher Bot
241+ User-agent: PodchaserParser
242+ User-agent: PostGooBot
243+ User-agent: PressEngine Bot
244+ User-agent: PressEngineBot
245+ User-agent: Pro Sitemaps
246+ User-agent: Pro-Sitemaps/
247+ User-agent: ProjectShield Url Check
248+ User-agent: ProjectShield-UrlCheck
249+ User-agent: Protopage
250+ User-agent: proximic
251+ User-agent: PubMatic
252+ User-agent: Qwantbot
253+ User-agent: Readybot.io
254+ User-agent: ReadYou
255+ User-agent: RecordedFuture Global Inventory Crawler
256+ User-agent: redditbot
257+ User-agent: retresco-tms
258+ User-agent: rss-monitor
259+ User-agent: rss2tg bot
260+ User-agent: scope3
261+ User-agent: Screaming Frog SEO Spider
262+ User-agent: search.marginalia.nu
263+ User-agent: Seeding Alliance ads.txt Check
264+ User-agent: Seekport
265+ User-agent: Seekport Crawler
266+ User-agent: SeekportBot
267+ User-agent: Sellers.Guide Crawler by Primis
268+ User-agent: semantic-visions
269+ User-agent: Seznam
270+ User-agent: SeznamBot
271+ User-agent: SimplePie
272+ User-agent: SinceraSyntheticUser
273+ User-agent: SISTRIX Crawler
274+ User-agent: SISTRIX Optimizer
275+ User-agent: Skype
276+ User-agent: SkypeUriPreview
277+ User-agent: Slack Image Proxy
278+ User-agent: Slack-ImgProxy
279+ User-agent: Slackbot
280+ User-agent: Slackbot 1.0
281+ User-agent: Slackbot-LinkExpanding
282+ User-agent: SlackbotLinkExpanding
283+ User-agent: SmartFeedBot
284+ User-agent: SmartologyBot
285+ User-agent: Snap URL Preview Service
286+ User-agent: SnapchatAds
287+ User-agent: SpaceCowboys RSS Reader
288+ User-agent: Speroll-AdsTxt-Crawler
289+ User-agent: StartmeBot
290+ User-agent: StatistikHessen
291+ User-agent: SteamChat
292+ User-agent: Storebot-Google
293+ User-agent: SummalyBot
294+ User-agent: Superfeedr
295+ User-agent: SurdotlyBot
296+ User-agent: Synapse
297+ User-agent: Telegram Bot
298+ User-agent: TelegramBot
299+ User-agent: theoldreader.com
300+ User-agent: Tickaroo-LinkPreview
301+ User-agent: TinEye-Web
302+ User-agent: Tiny Tiny RSS
303+ User-agent: tollbot
304+ User-agent: Trade Desk ads.txt
305+ User-agent: Trello Link Preview
306+ User-agent: TrueRelevanceBot
307+ User-agent: TTD-Content
308+ User-agent: Tumblr
309+ User-agent: Tumblr/
310+ User-agent: Twitterbot
311+ User-agent: Twitterbot/
312+ User-agent: uipbot
313+ User-agent: UrlDownloadabilityBot
314+ User-agent: Verity
315+ User-agent: Viber
316+ User-agent: ViennaTinyBot
317+ User-agent: ViralizeBot
318+ User-agent: vkShare
319+ User-agent: W3 Validator Services
320+ User-agent: W3C-checklink
321+ User-agent: w4mwnpbXf3MFAbxOkJRw
322+ User-agent: WebMoni/
323+ User-agent: WebsiteMetadataRetriever
324+ User-agent: WhatsApp
325+ User-agent: WMF Zotero Translation Server
326+ User-agent: WPMU DEV Broken Link Checker
327+ User-agent: Xing Bot
328+ User-agent: XingBot
329+ User-agent: yacybot
330+ User-agent: Yahoo Ad monitoring
331+ User-agent: Yahoo Link Preview
332+ User-agent: Yahoo! Slurp
333+ User-agent: YahooMailProxy
334+ User-agent: yandex
335+ User-agent: YandexBot
336+ User-agent: YandexImages
337+ User-agent: YandexMobileBot
338+ User-agent: YandexRenderResourcesBot
339+ User-agent: YandexVideo
340+ User-agent: ZetaGlobal-adstxt
341+ User-agent: ZoteroTranslationServer/WMF
342+ Allow: /
3343 
4- User-agent: *
5344  Disallow: /zeit/
6345  Disallow: /suche/
7346  Disallow: /templates/
@@ -14,108 +353,29 @@
14353  Disallow: /campus/framebuilder/
15354  Disallow: /navigation-teasers*
16355  Disallow: *iqadcontroller.js
17- Allow: /llms.txt
18-
19- User-agent: anthropic-ai
20- Disallow: /
21-
22- User-agent: Ai2Bot-Dolma
23- Disallow: /
24-
25- User-agent: Applebot-Extended
26- Disallow: /
27-
28- User-agent: Baiduspider
29- Disallow: /
30-
31- User-agent: Bytespider
32- Disallow: /
33-
34- User-agent: CCBot
35- Disallow: /
36-
37- User-agent: ChatGLM-Spider
38- Disallow: /
39-
40- User-agent: ClaudeBot
41- Disallow: /
42-
43- User-agent: CloudVertexBot
44- Disallow: /
45-
46- User-agent: cohere-training-data-crawler
47- Disallow: /
48-
49- User-agent: Cotoyogi
50- Disallow: /
51-
52- User-agent: DeepSeekBot
53- Disallow: /
54-
55- User-agent: Diffbot
56- Disallow: /
57-
58- User-agent: FacebookBot
59- Disallow: /
60-
61- User-agent: Google-CloudVertexBot
62- Disallow: /
356+ Allow: /
63357 
64358  User-agent: Google-Extended
65359  Disallow: /
66360  Allow: /*-gxe$
67361 
68- User-agent: GPTBot
69- Disallow: /
70-
71- User-agent: Google-Extended
72- Disallow: /
73- Allow: /*-gxe$
362+ User-agent: Googlebot-News
363+ Disallow: /angebote/
74364 
75365  User-agent: GrapeshotCrawler
76366  crawl-delay: 3
77367 
78- User-agent: img2dataset
368+ User-agent: *
79369  Disallow: /
80-
81- User-agent: Kangaroo Bot
82- Disallow: /
83-
84- User-agent: KunatoCrawler
85- Disallow: /
86-
87- User-agent: Meta-ExternalAgent
88- Disallow: /
89-
90- User-agent: PanguBot
91- Disallow: /
92-
93- User-agent: Perplexity-User
94- Disallow: /
95-
96- User-agent: PerplexityBot
97- Disallow: /
98-
99- User-agent: quillbot.com
100- Disallow: /
101-
102- User-agent: Spider
103- Disallow: /
104-
105- User-agent: TerraCotta
106- Disallow: /
107-
108- User-agent: Timpibot
109- Disallow: /
110-
111- User-agent: VelenPublicWebCrawler
112- Disallow: /
113-
370+ Allow:/llms.txt
114371 
115372  Sitemap: https://www.zeit.de/gsitemaps/index.xml
116373 
117- # Legal notice: zeit.de expressly reserves the right to use its content for commercial text and data mining (§ 44 b UrhG).
118- # The use of robots or other automated means to access zeit.de or collect or mine data without
119- # the express permission of zeit.de is strictly prohibited.
120- # zeit.de may, in its discretion, permit certain automated access to certain zeit.de pages,
121- # If you would like to apply for permission to crawl zeit.de, collect or use data, please email [email protected].
374+ # Legal notice: zeit.de expressly reserves the right to use its content
375+ # for commercial text and data mining (§ 44 b UrhG).
376+ # The use of robots or other automated means to access zeit.de or collect
377+ # or mine data without the express permission of zeit.de is strictly prohibited.
378+ # zeit.de may, in its discretion, permit certain automated access to certain
379+ # zeit.de pages.
380+ # If you would like to apply for permission to crawl zeit.de, collect or use
381+ # data, please email [email protected].