NeuralCrawl

Die Zeit / robots.txt snapshot

← back to zeit.de · fetched 2026-11-09T15:07:03Z (9h ago) · HTTP 200 · 10251 bytes · sha256 5a651f19fa463033 · raw

final URL: https://www.zeit.de/robots.txt

1User-agent: AASA-Bot
2User-agent: Accessible Web Bot
3User-agent: AccessibleWebBot
4User-agent: AdDefend
5User-agent: adidxbot
6User-agent: AdkernelTopicCrawler
7User-agent: AdsBot-Google
8User-agent: AdsBot-Google-Mobile
9User-agent: Adstxt Parser, Demandbase
10User-agent: AdsTxtCrawler
11User-agent: Akkoma
12User-agent: Amazon AdBot
13User-agent: Amazon-Advertising-ad-standards-bot
14User-agent: AmazonAdBot
15User-agent: APIs-Google
16User-agent: Apple AuthenticationServicesAgent
17User-agent: Apple Link Preview
18User-agent: Applebot
19User-agent: AppleNewsBot
20User-agent: ArachnysSpider
21User-agent: Assaybot
22User-agent: Automattic
23User-agent: Awario
24User-agent: AwarioBot
25User-agent: AwarioRssBot
26User-agent: AwarioSmartBot
27User-agent: BazQux
28User-agent: Bing Ads
29User-agent: Bing Preview
30User-agent: bingbot
31User-agent: bingbot/
32User-agent: BingPreview
33User-agent: bitlybot
34User-agent: Bluesky
35User-agent: botify
36User-agent: BrandVerity
37User-agent: brokenlinkcheck
38User-agent: Brusselsbot
39User-agent: Buck/
40User-agent: Buckbot
41User-agent: BufferLinkPreviewBot
42User-agent: captify-crawler
43User-agent: Chrome Privacy Preserving Prefetch Proxy
44User-agent: Chrome-Lighthouse
45User-agent: citoid
46User-agent: Citoid/WMF
47User-agent: CommaFeed
48User-agent: compatible; proximic
49User-agent: Country Feed Monitor
50User-agent: CriteoBot
51User-agent: CriteoBot/
52User-agent: Cxense
53User-agent: cXensebot
54User-agent: cXensebot/
55User-agent: deadlinkchecker
56User-agent: Discordbot
57User-agent: dlvr.it
58User-agent: dlvrit
59User-agent: DobbyNewsBot
60User-agent: Domains
61User-agent: domains-distillery
62User-agent: DoubleVerifyBot
63User-agent: DuckDuckBot
64User-agent: DVbot
65User-agent: Echobot Bot
66User-agent: Edge Health Probe
67User-agent: ElevenlabsBot
68User-agent: ElevenlabsCrawler
69User-agent: Embedly
70User-agent: emetriqContextualBot
71User-agent: EventRadar/1.0
72User-agent: ExaUser
73User-agent: ExteAdstxtCrawl
74User-agent: facebookcatalog
75User-agent: facebookexternalhit
76User-agent: facebookexternalua
77User-agent: facebookplatform
78User-agent: Facebot
79User-agent: FairOpenNewsBot
80User-agent: FAST-WebCrawler
81User-agent: Fastmail Bot
82User-agent: FastmailUA
83User-agent: Feedbin
84User-agent: feedbot
85User-agent: FeedBurner
86User-agent: Feeder /
87User-agent: feeder.co
88User-agent: FeedFetcher-Google
89User-agent: FeedFlow
90User-agent: Feedler/1.0
91User-agent: Feedly
92User-agent: Feedly/1
93User-agent: Feedsearch
94User-agent: Feedsearch-Crawler
95User-agent: FeedValidator
96User-agent: FleebsBot
97User-agent: FlipboardProxy
98User-agent: FlipboardRSS
99User-agent: FreshRSS
100User-agent: Friendica
101User-agent: Gettrbot
102User-agent: GlancelotBot
103User-agent: GnowitNewsbot
104User-agent: GoodNews-Crawler
105User-agent: Google AdsBot
106User-agent: Google AdSense
107User-agent: Google AppsViewer
108User-agent: Google Digital Asset Links
109User-agent: Google Favicon
110User-agent: Google Feed Fetcher
111User-agent: Google Image Proxy
112User-agent: Google Images
113User-agent: Google Inspection Tool
114User-agent: Google Messages Link Preview
115User-agent: Google Publisher Center
116User-agent: Google Schema Markup Testing Tool
117User-agent: Google Scholar
118User-agent: Google StoreBot
119User-agent: Google Videos
120User-agent: Google Web Preview
121User-agent: Google-Ads-Creatives-Assistant
122User-agent: Google-adstxt
123User-agent: Google-AdWords-Express
124User-agent: Google-Adwords-Instant
125User-agent: Google-BusinessLinkVerification
126User-agent: Google-Calendar-Importer
127User-agent: Google-Certificates-Bridge
128User-agent: Google-Display-Ads-Bot
129User-agent: Google-Flights-Search-Spackle
130User-agent: Google-InspectionTool
131User-agent: Google-NotebookLM
132User-agent: Google-PageRenderer
133User-agent: Google-PhysicalWeb
134User-agent: Google-Read-Aloud
135User-agent: Google-Safety
136User-agent: Google-Site-Verification
137User-agent: Google-Sitemaps
138User-agent: Google-Structured-Data-Testing
139User-agent: Google-Structured-Data-Testing-Tool
140User-agent: Google-TravelAds-Live
141User-agent: Google-Trust-Services
142User-agent: google-xrawler
143User-agent: google.com/feedfetcher
144User-agent: Google_InspectionTool
145User-agent: GoogleAssociationService
146User-agent: Googlebot
147User-agent: Googlebot-IA
148User-agent: Googlebot-IA/
149User-agent: Googlebot-Image
150User-agent: Googlebot-Mobile
151User-agent: Googlebot-News
152User-agent: Googlebot-Video
153User-agent: GoogleDocs
154User-agent: GoogleImageProxy
155User-agent: GoogleOther
156User-agent: GoogleOther-Image
157User-agent: GoogleOther-Video
158User-agent: GoogleProducer
159User-agent: GoogleStackdriverMonitoring-UptimeChecks
160User-agent: GoogleUserTriggeredFetcher
161User-agent: GrapeshotCrawler
162User-agent: Greppr
163User-agent: GumGum
164User-agent: Hootsuite-Authoring
165User-agent: HubSpot
166User-agent: HubSpot Crawler
167User-agent: IAB-Tech-Lab
168User-agent: ias-ie
169User-agent: ias-or
170User-agent: ias-va
171User-agent: ias_crawler
172User-agent: Iframely
173User-agent: IFTTT
174User-agent: Inoreader
175User-agent: inoreader.com
176User-agent: InteractiveAdvertisingBureau
177User-agent: IServ News Bot
178User-agent: Jetslide
179User-agent: JournalistAppBot
180User-agent: Leikibot
181User-agent: LetsEncrypt
182User-agent: LinkedInBot
183User-agent: Magus
184User-agent: Marginalia Search
185User-agent: Mastodon
186User-agent: Mediapartners-Google
187User-agent: meta-externalads
188User-agent: MetaInspector
189User-agent: MetaJobBot
190User-agent: Microsoft News Ingestion
191User-agent: Microsoft Preview
192User-agent: MicrosoftPreview
193User-agent: MicrosoftPreview/
194User-agent: Miniflux
195User-agent: MJ12bot
196User-agent: MochiImageCrawler
197User-agent: Monsidobot
198User-agent: MS Search 4.0 Robot
199User-agent: MS Search 6.0 Robot
200User-agent: MSMNewsBot
201User-agent: msnbot
202User-agent: msnbot-media
203User-agent: MuckRack
204User-agent: NanoInteractive
205User-agent: NetNewsWire
206User-agent: Netvibes
207User-agent: NewsBlur
208User-agent: NewsCrawler
209User-agent: Newslitbot
210User-agent: NewsNow
211User-agent: Newstral
212User-agent: NewsVergleichDE
213User-agent: NewsWallBot
214User-agent: NewzBot
215User-agent: NextCloud
216User-agent: Nextcloud Server Crawler
217User-agent: Nextcloud-Server-Crawler
218User-agent: NoahBot
219User-agent: Nooshub
220User-agent: OdklBot/
221User-agent: officestorebot
222User-agent: OpenGraph.io
223User-agent: OpenGraphBot
224User-agent: OpenindexSpider
225User-agent: outbrain
226User-agent: Overcast
227User-agent: Overcast/1.0 Podcast Sync
228User-agent: Ozon Web Grabber
229User-agent: Paqlebot
230User-agent: peer39_crawler
231User-agent: peer39_crawler/1.0
232User-agent: PetalBot
233User-agent: Pingdom
234User-agent: Pinterest
235User-agent: pinterest.com/
236User-agent: Pinterestbot
237User-agent: Pleroma
238User-agent: Pocket Casts Feed Parser
239User-agent: PocketImageCache
240User-agent: Podcatcher Bot
241User-agent: PodchaserParser
242User-agent: PostGooBot
243User-agent: PressEngine Bot
244User-agent: PressEngineBot
245User-agent: Pro Sitemaps
246User-agent: Pro-Sitemaps/
247User-agent: ProjectShield Url Check
248User-agent: ProjectShield-UrlCheck
249User-agent: Protopage
250User-agent: proximic
251User-agent: PubMatic
252User-agent: Qwantbot
253User-agent: Readybot.io
254User-agent: ReadYou
255User-agent: RecordedFuture Global Inventory Crawler
256User-agent: redditbot
257User-agent: retresco-tms
258User-agent: rss-monitor
259User-agent: rss2tg bot
260User-agent: scope3
261User-agent: Screaming Frog SEO Spider
262User-agent: search.marginalia.nu
263User-agent: Seeding Alliance ads.txt Check
264User-agent: Seekport
265User-agent: Seekport Crawler
266User-agent: SeekportBot
267User-agent: Sellers.Guide Crawler by Primis
268User-agent: semantic-visions
269User-agent: Seznam
270User-agent: SeznamBot
271User-agent: SimplePie
272User-agent: SinceraSyntheticUser
273User-agent: SISTRIX Crawler
274User-agent: SISTRIX Optimizer
275User-agent: Skype
276User-agent: SkypeUriPreview
277User-agent: Slack Image Proxy
278User-agent: Slack-ImgProxy
279User-agent: Slackbot
280User-agent: Slackbot 1.0
281User-agent: Slackbot-LinkExpanding
282User-agent: SlackbotLinkExpanding
283User-agent: SmartFeedBot
284User-agent: SmartologyBot
285User-agent: Snap URL Preview Service
286User-agent: SnapchatAds
287User-agent: SpaceCowboys RSS Reader
288User-agent: Speroll-AdsTxt-Crawler
289User-agent: StartmeBot
290User-agent: StatistikHessen
291User-agent: SteamChat
292User-agent: Storebot-Google
293User-agent: SummalyBot
294User-agent: Superfeedr
295User-agent: SurdotlyBot
296User-agent: Synapse
297User-agent: Telegram Bot
298User-agent: TelegramBot
299User-agent: theoldreader.com
300User-agent: Tickaroo-LinkPreview
301User-agent: TinEye-Web
302User-agent: Tiny Tiny RSS
303User-agent: tollbot
304User-agent: Trade Desk ads.txt
305User-agent: Trello Link Preview
306User-agent: TrueRelevanceBot
307User-agent: TTD-Content
308User-agent: Tumblr
309User-agent: Tumblr/
310User-agent: Twitterbot
311User-agent: Twitterbot/
312User-agent: uipbot
313User-agent: UrlDownloadabilityBot
314User-agent: Verity
315User-agent: Viber
316User-agent: ViennaTinyBot
317User-agent: ViralizeBot
318User-agent: vkShare
319User-agent: W3 Validator Services
320User-agent: W3C-checklink
321User-agent: w4mwnpbXf3MFAbxOkJRw
322User-agent: WebMoni/
323User-agent: WebsiteMetadataRetriever
324User-agent: WhatsApp
325User-agent: WMF Zotero Translation Server
326User-agent: WPMU DEV Broken Link Checker
327User-agent: Xing Bot
328User-agent: XingBot
329User-agent: yacybot
330User-agent: Yahoo Ad monitoring
331User-agent: Yahoo Link Preview
332User-agent: Yahoo! Slurp
333User-agent: YahooMailProxy
334User-agent: yandex
335User-agent: YandexBot
336User-agent: YandexImages
337User-agent: YandexMobileBot
338User-agent: YandexRenderResourcesBot
339User-agent: YandexVideo
340User-agent: ZetaGlobal-adstxt
341User-agent: ZoteroTranslationServer/WMF
342Allow: /
343
344Disallow: /zeit/
345Disallow: /suche/
346Disallow: /templates/
347Disallow: /hp_channels/
348Disallow: /send/
349Disallow: /rezepte/suche/
350Disallow: */comment-thread?
351Disallow: */liveblog-backend*
352Disallow: /framebuilder/
353Disallow: /campus/framebuilder/
354Disallow: /navigation-teasers*
355Disallow: *iqadcontroller.js
356Allow: /
357
358User-agent: Google-Extended
359Disallow: /
360Allow: /*-gxe$
361
362User-agent: Googlebot-News
363Disallow: /angebote/
364
365User-agent: GrapeshotCrawler
366crawl-delay: 3
367
368User-agent: *
369Disallow: /
370Allow:/llms.txt
371
372Sitemap: https://www.zeit.de/gsitemaps/index.xml
373
374# Legal notice: zeit.de expressly reserves the right to use its content
375# for commercial text and data mining (§ 44 b UrhG).
376# The use of robots or other automated means to access zeit.de or collect
377# or mine data without the express permission of zeit.de is strictly prohibited.
378# zeit.de may, in its discretion, permit certain automated access to certain
379# zeit.de pages.
380# If you would like to apply for permission to crawl zeit.de, collect or use
381# data, please email [email protected].