NeuralCrawl

De Telegraaf / robots.txt snapshot

← back to telegraaf.nl · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 2656 bytes · sha256 e12e0c1acebf9301 · raw

final URL: https://www.telegraaf.nl/robots.txt

1# Generated: 2026-06-18T09:55:50.306Z
2# Brand: telegraaf.nl | Environment: prod
3
4# All copyrights, neighbouring rights and database rights in the content and layout of this website/app are explicitly reserved and are for personal, non-commercial use only.
5# In accordance with Article 4 of the Directive on Copyright in the Digital Single Market (CDSM) and its transposition into the law of the applicable Member State,
6# all content of this website on which it is made available is not to be used for the purposes of text and data mining, extraction, scraping and/or the use of programs or robots
7# for automatic data collection and/or extraction of digital data, whether for machine learning or artificial intelligence purposes or otherwise.
8# See also the Terms and Conditions of this website.
9
10# robots.txt prod De Telegraaf
11user-agent: *
12Allow: /
13Allow: /tags
14
15# Disallow Internal Search
16Disallow: /zoeken/
17
18# Disallow bundle
19Disallow: /*?bundle
20
21# Disallow infront widgets
22Disallow: /infront/widget/
23
24# Disallow Sponsored Articles for Google News
25User-agent: Googlebot-News
26Disallow: /branded-content/
27Disallow: /brandedcontent/
28
29# Disallow Large Language Models
30User-agent: Amazonbot
31Disallow: /
32
33User-agent: anthropic-ai
34Disallow: /
35
36User-agent: Bytespider
37Disallow: /
38
39User-agent: CCBot
40Disallow: /
41
42User-agent: ChatGPT-User
43Disallow: /
44
45User-agent: ClaudeBot
46Disallow: /
47
48User-agent: Claude-Web
49Disallow: /
50
51User-agent: cohere-ai
52Disallow: /
53
54User-agent: Diffbot
55Disallow: /
56
57User-agent: FacebookBot
58Disallow: /
59
60User-agent: Google-Extended
61Disallow: /
62
63User-agent: GPTBot
64Disallow: /
65
66User-agent: magpie-crawler
67Disallow: /
68
69User-agent: omgili
70Disallow: /
71
72User-agent: omgilibot
73Disallow: /
74
75User-agent: PerplexityBot
76Disallow: /
77
78User-agent: Google-CloudVertexBot
79Disallow: /
80
81User-agent: meta-externalagent
82Disallow: /
83
84User-agent: meta-externalfetcher
85Disallow: /
86
87User-agent: Ahrefsbot
88Disallow: /
89
90User-agent: Archive.org_bot
91Disallow: /
92
93User-agent: Bravebot
94Disallow: /
95
96User-agent: Claude-Searchbot
97Disallow: /
98
99User-agent: Claude-User
100Disallow: /
101
102User-agent: DeepSeekBot
103Disallow: /
104
105User-agent: Meta-WebIndexer
106Disallow: /
107
108User-agent: MistralAI-Index
109Disallow: /
110
111User-agent: MistralAI-User
112Disallow: /
113
114User-agent: OAI-AdsBot
115Disallow: /
116
117User-agent: OAI-Searchbot
118Disallow: /
119
120User-agent: Perplexity-User
121Disallow: /
122
123User-agent: Youbot
124Disallow: /
125
126# User-agent: Bingbot
127# Disallow: /
128
129#list sitemaps
130Sitemap: https://www.telegraaf.nl/sitemap.xml
131Sitemap: https://www.telegraaf.nl/sitemap-image.xml
132Sitemap: https://www.telegraaf.nl/sitemap-news.xml
133Sitemap: https://www.telegraaf.nl/sitemap-video.xml
134
135# Served via new CDN