NeuralCrawl

Germany (Bundesregierung) / robots.txt snapshot

← back to bundesregierung.de · fetched 2026-06-20T11:49:11Z (7h ago) · HTTP 200 · 1935 bytes · sha256 be4ec6b2f577b156 · raw

final URL: manual:file

1User-agent: AhrefsBot
2Disallow: /
3
4User-agent: AlphaBot
5Disallow: /
6
7User-agent: Baiduspider
8Disallow: /
9
10User-agent: Baiduspider-render
11Disallow: /
12
13User-agent: Buck
14Disallow: /
15
16User-agent: changedetection
17Disallow: /
18
19User-agent: Cliqzbot
20Disallow: /
21
22User-agent: Exabot
23Disallow: /
24
25User-agent: Flamingo_SearchEngine
26Disallow: /
27
28User-agent: Grobbot
29Disallow: /
30
31User-agent: JobboerseBot
32Disallow: /
33
34User-agent: jobs.de-Robot
35Disallow: /
36
37User-agent: linkdexbot
38Disallow: /
39
40User-agent: Mail.RU_Bot
41Disallow: /
42
43User-agent: Mediatoolkitbot
44Disallow: /
45
46User-agent: MegaIndex.ru/2.0
47Disallow: /
48
49User-agent: Mega-Index
50Disallow: /
51
52User-agent: MegaIndex.ru
53Disallow: /
54
55User-agent: MJ12bot
56Disallow: /
57
58User-agent: MojeekBot
59Disallow: /
60
61User-agent: Pinterestbot
62Disallow: /
63
64User-agent: rogerbot
65Disallow: /
66
67User-agent: SafeDNSBot
68Disallow: /
69
70User-agent: SemrushBot
71Disallow: /
72
73User-agent: SemrushBot/2~bl
74Disallow: /
75
76User-agent: SemrushBot-BA
77Disallow: /
78
79User-agent: SemrushBot-BM
80Disallow: /
81
82User-agent: SemrushBot-SA
83Disallow: /
84
85User-agent: SEOkicks-Robot
86Disallow: /
87
88User-agent: SeznamBot
89Disallow: /
90
91User-agent: SMTBot
92Disallow: /
93
94User-agent: spbot
95Disallow: /
96
97User-agent: trendictionbot
98Disallow: /
99
100User-agent: TweetmemeBot
101Disallow: /
102
103User-agent: vebidoobot
104Disallow: /
105
106User-agent: wonderbot
107Disallow: /
108
109User-agent: Wotbox
110Disallow: /
111
112User-agent: yacybot
113Disallow: /
114
115User-agent: YaK
116Disallow: /
117
118User-agent: YandexAntivirus
119Disallow: /
120
121User-agent: YandexBot/3.0
122Disallow: /
123
124User-agent: YandexImages
125Disallow: /
126
127User-agent: YandexMobileBot
128Disallow: /
129
130User-agent: Yeti
131Disallow: /
132
133Sitemap: https://www.bundesregierung.de/service-sitemap-4d4e784d6cdbde1a37901f96601df03f-sitemap_index.xml
134Sitemap: https://www.bundesregierung.de/service-sitemap-eac0bb2dddac4321ff3b164c4ce8d951-sitemap_index.xml
135Sitemap: https://www.bundesregierung.de/service-sitemap-9dca52ff98e0ed01b98a219a2b0bb383-sitemap_index.xml