NeuralCrawl

Bild / robots.txt snapshot

← back to bild.de · fetched 2026-08-24T13:15:37Z (26d ago) · HTTP 200 · 3053 bytes · sha256 f2af535c9e54f181 · raw

final URL: https://www.bild.de/robots.txt

1License: https://www.bild.de/rsl.xml
2
3User-agent: AhrefsBot
4User-agent: AI2Bot
5User-agent: Ai2Bot-Dolma
6User-agent: aiHitBot
7User-agent: Amazonbot
8User-agent: Andibot
9User-agent: Applebot-Extended
10User-agent: Awario
11User-agent: Baiduspider
12User-agent: bedrockbot
13User-agent: BLEXBot
14User-agent: Brightbot 1.0
15User-agent: Bytespider
16User-agent: CCBot
17User-agent: cohere-ai
18User-agent: cohere-training-data-crawler
19User-agent: Cotoyogi
20User-agent: Crawlspace
21User-agent: DataForSeoBot
22User-agent: Datenbank Crawler
23User-agent: Devin
24User-agent: Diffbot
25User-agent: dotbot
26User-agent: DuckAssistBot
27User-agent: Echobot Bot
28User-agent: EchoboxBot
29User-agent: Factset_spyderbot
30User-agent: FirecrawlAgent
31User-agent: Flamingo_SearchEngine
32User-agent: FriendlyCrawler
33User-agent: ia_archiver
34User-agent: iaskspider/2.0
35User-agent: ICC-Crawler
36User-agent: ImagesiftBot
37User-agent: img2dataset
38User-agent: ISSCyberRiskCrawler
39User-agent: Kangaroo Bot
40User-agent: LCC
41User-agent: magpie-crawler
42User-agent: MistralAI-User
43User-agent: MistralAI-User/1.0
44User-agent: MJ12bot
45User-agent: MyCentralAIScraperBot
46User-agent: netEstate Imprint Crawler
47User-agent: NovaAct
48User-agent: omgili
49User-agent: omgilibot
50User-agent: Operator
51User-agent: PanguBot
52User-agent: Panscient
53User-agent: panscient.com
54User-agent: Perplexity-User
55User-agent: PerplexityBot
56User-agent: PetalBot
57User-agent: PhindBot
58User-agent: Poseidon Research Crawler
59User-agent: QualifiedBot
60User-agent: QuillBot
61User-agent: quillbot.com
62User-agent: SBIntuitionsBot
63User-agent: Scrapy
64User-agent: SemrushBot
65User-agent: SemrushBot-OCOB
66User-agent: SemrushBot-SA
67User-agent: SemrushBot-SWA
68User-agent: sentibot
69User-agent: Sidetrade indexer bot
70User-agent: SirdataBot
71User-agent: sogou spider
72User-agent: SummalyBot
73User-agent: TerraCotta
74User-agent: Thinkbot
75User-agent: TikTokSpider
76User-agent: Timpibot
77User-agent: TurnitinBot
78User-agent: VelenPublicWebCrawler
79User-agent: WARDBot
80User-agent: Webzio-Extended
81User-agent: wpbot
82User-agent: YandexAdditional
83User-agent: YandexAdditionalBot
84User-agent: Yeti
85User-agent: YouBot
86Disallow: /
87
88User-agent: Googlebot-News
89Disallow: /partner/
90Disallow: /jobs/api/gmapi
91Disallow: /community/
92Disallow: /sonstiges/vorproduktion/
93Disallow: /test_stage/
94Disallow: /test/
95Disallow: /energieloesungen/*-form/
96Disallow: /energieloesungen/social/*-form/
97Disallow: /energieloesungen/vergleich/*-form/
98Disallow: /leben-im-alter/*-form/
99Disallow: /cmsid/
100
101User-agent: *
102Disallow: /jobs/api/gmapi
103Disallow: /community/
104Disallow: /sonstiges/vorproduktion/
105Disallow: /test_stage/
106Disallow: /test/
107Disallow: /energieloesungen/*-form/
108Disallow: /energieloesungen/social/*-form/
109Disallow: /energieloesungen/vergleich/*-form/
110Disallow: /leben-im-alter/*-form/
111Disallow: /cmsid/
112Disallow: /buehne/
113Disallow: /sonstiges/intern/
114Disallow: /vorprod/
115Disallow: /vorproduktion/
116
117# Sitemaps
118Sitemap: https://www.bild.de/sitemap-news.xml
119Sitemap: https://www.bild.de/sitemap-video.xml
120Sitemap: https://www.bild.de/sitemap-index.xml
121Sitemap: https://www.bild.de/sitemap-category.xml