NeuralCrawl

Bild / robots.txt snapshot

← back to bild.de · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 3349 bytes · sha256 851df81a57c1ed99 · raw

final URL: https://www.bild.de/robots.txt

1License: https://www.bild.de/rsl.xml
2
3User-agent: AhrefsBot
4User-agent: AI2Bot
5User-agent: Ai2Bot-Dolma
6User-agent: aiHitBot
7User-agent: Andibot
8User-agent: Applebot-Extended
9User-agent: Awario
10User-agent: Baiduspider
11User-agent: bedrockbot
12User-agent: BLEXBot
13User-agent: Brightbot 1.0
14User-agent: Bytespider
15User-agent: CCBot
16User-agent: cohere-ai
17User-agent: cohere-training-data-crawler
18User-agent: Cotoyogi
19User-agent: Crawlspace
20User-agent: DataForSeoBot
21User-agent: Datenbank Crawler
22User-agent: Devin
23User-agent: Diffbot
24User-agent: dotbot
25User-agent: DuckAssistBot
26User-agent: Echobot Bot
27User-agent: EchoboxBot
28User-agent: Factset_spyderbot
29User-agent: FirecrawlAgent
30User-agent: Flamingo_SearchEngine
31User-agent: FriendlyCrawler
32User-agent: ia_archiver
33User-agent: iaskspider/2.0
34User-agent: ICC-Crawler
35User-agent: ImagesiftBot
36User-agent: img2dataset
37User-agent: ISSCyberRiskCrawler
38User-agent: Kangaroo Bot
39User-agent: LCC
40User-agent: magpie-crawler
41User-agent: MistralAI-User
42User-agent: MistralAI-User/1.0
43User-agent: MJ12bot
44User-agent: MyCentralAIScraperBot
45User-agent: netEstate Imprint Crawler
46User-agent: NovaAct
47User-agent: omgili
48User-agent: omgilibot
49User-agent: Operator
50User-agent: PanguBot
51User-agent: Panscient
52User-agent: panscient.com
53User-agent: Perplexity-User
54User-agent: PerplexityBot
55User-agent: PetalBot
56User-agent: PhindBot
57User-agent: Poseidon Research Crawler
58User-agent: QualifiedBot
59User-agent: QuillBot
60User-agent: quillbot.com
61User-agent: SBIntuitionsBot
62User-agent: Scrapy
63User-agent: SemrushBot
64User-agent: SemrushBot-OCOB
65User-agent: SemrushBot-SA
66User-agent: SemrushBot-SWA
67User-agent: sentibot
68User-agent: Sidetrade indexer bot
69User-agent: SirdataBot
70User-agent: sogou spider
71User-agent: SummalyBot
72User-agent: TerraCotta
73User-agent: Thinkbot
74User-agent: TikTokSpider
75User-agent: Timpibot
76User-agent: TurnitinBot
77User-agent: VelenPublicWebCrawler
78User-agent: WARDBot
79User-agent: Webzio-Extended
80User-agent: wpbot
81User-agent: YandexAdditional
82User-agent: YandexAdditionalBot
83User-agent: Yeti
84User-agent: YouBot
85Disallow: /
86
87User-agent: Googlebot-News
88Disallow: /partner/
89Disallow: /dating/go/
90Disallow: /games/go/
91Disallow: /casinos/go/
92Disallow: /online-apotheke/go/
93Disallow: /jobs/api/gmapi
94Disallow: /community/
95Disallow: /vorprod/
96Disallow: /test_stage/
97Disallow: /test/
98Disallow: /energieloesungen/*-form/
99Disallow: /energieloesungen/go/
100Disallow: /energieloesungen/social/*-form/
101Disallow: /energieloesungen/social/go/
102Disallow: /energieloesungen/vergleich/*-form/
103Disallow: /energieloesungen/vergleich/go/
104Disallow: /leben-im-alter/*-form/
105Disallow: /leben-im-alter/go/
106
107User-agent: *
108Disallow: /dating/go/
109Disallow: /games/go/
110Disallow: /casinos/go/
111Disallow: /online-apotheke/go/
112Disallow: /jobs/api/gmapi
113Disallow: /community/
114Disallow: /vorprod/
115Disallow: /test_stage/
116Disallow: /test/
117Disallow: /energieloesungen/*-form/
118Disallow: /energieloesungen/go/
119Disallow: /energieloesungen/social/*-form/
120Disallow: /energieloesungen/social/go/
121Disallow: /energieloesungen/vergleich/*-form/
122Disallow: /energieloesungen/vergleich/go/
123Disallow: /leben-im-alter/*-form/
124Disallow: /leben-im-alter/go/
125
126# Sitemaps
127Sitemap: https://www.bild.de/sitemap-news.xml
128Sitemap: https://www.bild.de/sitemap-video.xml
129Sitemap: https://www.bild.de/sitemap-index.xml
130Sitemap: https://www.bild.de/sitemap-category.xml